Dr. Deborah Swain
AI Lab Report
For my Artificial Intelligence Lab Report, I will be critiquing a thesis project by MIT student José Humberto Espinosa Chrislieb, titled “Reducing Complexity of Consumer Electronics Interfaces Using Common Sense Reasoning.” The abstract of the thesis states that the project is based around the idea that consumer user electronics require far too much input from devices and generally too many devices to control. The project asserts that devices such as microwaves, phones, and cameras are becoming too complicated for the average user.
The thesis gears itself towards generating a natural language-based system or process that will inherently understand and predict what the user is attempting to do. Not only this, but the system will be capable of troubleshooting and self-healing. The thesis proposes ROADIE, a three-tiered system of control that is “goal-oriented,” “self-describing,” “self-revealing,” “self-debugging,” and works on “common sense knowledge” (Chrislieb 2005;18-20).
Within this paper, I will judge the technology, applicability, status, and potential of this project on a 25 point scale respective to each of the four aforementioned themes. To begin, I will explain the system in as much depth as possible, without taking up the 100 pages the thesis consisted of.
The project by Mr. Chrislieb created a software-based controlling device called ROADIE. The paper does not go into detail on what ROADIE actually stands for. ROADIE is comprised of two parts: the user interaction module and the device controller. The user interaction module uses a system called EventNet to map the goals and actions of users into a language that the planner can understand. Without going into too many details, EventNet is modeled on the OpenMind Commonsense Project. This project is based in a web-interface where users submit data into templates to populate commonsense actions and responses. Here is an example the thesis provides: “The effect of walking in the rain” IS “getting wet” (Chrislieb 2005;23). As of 2005 (when the thesis was submitted) OpenMind had accumulated some 750,000 sentences. From OpenMind came ConceptNet, which mined OpenMind templates and used Natural Language Processing techniques. The project LifeNet then transformed this concept into an egocentric system. Under this system, the example given above would become this: I walk in the rain; I get wet (Chrislieb 2005;25). LifeNet also took the idea one step further by attempting to predict what the next logical process would be by introducing “probabilistic reasoning.” Finally, this brings us to EventNet which, as the thesis best explains:
EventNet uses the temporal nodes in LifeNet to create an association network. It can make predictions of the events most likely to occur before or after a certain set of events, in contrast to LifeNet’s single-event predictions. Also, it provides paths between nodes providing a plausible sequence of partially-ordered events occurring between two given events. It is able to infer that in order to watch a movie it is necessary to buy a ticket and that a person is likely to buy popcorn. (Chrislieb 2005;23-24)
ROADIE operates by using the User Interface, the Planner, and the Common Sense Reasoning System (EventNet). Here we can see the proposed system: (Chrislieb 2005;30)
Ultimately, the system utilizes EventNet to process reasonable actions that are generated from both the user’s input and the planner’s goals and requirements. The goal of the system is to enable the user to access multiple devices, obtain their states of being, and manipulate different devices to do whatever tasks are required. If a device is unable to perform a task, a good system should be able to propose an alternative, or give a reason that the inability occurred. The use of EventNet’s natural language makes it much easier and more accessible to the user in terms of debugging and overall maintenance.
Mr. Chrislieb observes that devices at this current time do not meet the requirements to “provide means to control their functions, and…query their state by external software” (Chrislieb 2005;31). For this reason, devices are created in a simulated environment.
The research on this particular thesis seems to conclude at the end of the paper, although Mr. Chrislieb does go into details about future applicability of the system. He tests ROADIE on 12 users, providing some additional data regarding the tests. This ends the discussion of the overall project goals and research. The next section will begin the ratings of the four components, technology, applicability, status, and potential.
In this section, we will go over the technology of the system. I have given the technology of the system an overall fifteen (15) points. It received an outstanding ten points for using an open source solution. EventNet is based on several different solutions, but they all hearken back to the OpenMind Commonsense Project (OCP). OCP relied on a volunteer based, user-supplied database of causes and their effects. I am a strong supporter of the open source; I think that it both enables creativity and bolsters adaptability to various situations.
However, in the same token, open source can also lead to problems. For this, I have taken away two and a half of the overall 25 points. While open source can provide a whole slew of benefits for the user, it can also cause a few problems. The thesis states in several places that using an off-shoot of OpenMind caused a few unexpected results. Often times, ROADIE would come up with inconsistent and downright incorrect responses to an action. This can mainly be explained by using volunteers to populate the OpenMind database.
Another issue I took with ROADIE was the exclusion of actual devices. I understand that devices are not presently adaptable to this kind of technology–the kind that can readily deploy its state to the interface or self-debug. However, it did not seem to me as I read the thesis that much of an attempt was made to interact with actual devices. The thesis even states that this became an issue as users attempted to use the system. Users often found it difficult to test a system with theoretical devices. It can be difficult for users to proceed when the system tells them to place a CD into the DVD player and move on to the next step when the DVD player does not really exist. For this, I deducted two and a half points.
On the other side of this, I was impressed with ROADIE’s ability to create a simulated system. Even though it would have been easier to have real devices to connect to, the author did create a functional simulated environment. Five points were awarded for the creativity to employ a simulation, rather than hypothesizing on the entire system and simply creating a “what-if” project.
‘Applicability’ refers to how and for whom the application (in this case ROADIE) will be utilized in real life. I gave the overall applicability a rating of ten. It was hard for me to separate applicability from potential. However, the way I distinguished the two ideas was to think of applicability as literally what the project did at the time it was implemented, and not what it was capable of doing in the future. In this regard, it garnered a lower score.
Applicability suffers from most of the same problems as technology. For one, the simulations simply made it difficult for many of the users. The author himself admitted as much. It also seemed to me that while ROADIE was intended to have far-reaching impacts and the ability to affect many different devices, the author focused solely on a few multimedia devices.
Speakers, DVD, TV, Keyboard? (Chrislieb 2005;55)
We use far more than a television, DVD player, stereo, and MIDI controller on a daily basis. How appropriate is the keyboard for the simulation? The author also never really explains who the users are that tested the system. Are they all media students? I don’t own a keyboard, and honestly do not know anyone who does. It seemed to me that a computer- or Internet-enabled device would have been far more appropriate. I deducted fifteen points overall for these flaws. Ten points were awarded, however, for the fairly exhaustive specific actions to be performed on the aforementioned four devices.
Because this was a thesis paper and not an ongoing project per se, I am going to go over the status of what occurred in the experiment and in the paper. This is ultimately a closed and finished project. The author does go into detail with respect to prior work on similar subjects and projects that are currently ongoing. For this, I awarded five points. It is very easy to access the end of the thesis paper and find a plethora of information on related ideas and topics.
The second area of award is in the thoroughness of the research. At the conclusion of the experiment, the author performed a great deal of analysis. From calculating the standard deviation of the overall understanding of ROADIE by his participants to rating the ease of use for configuring certain devices, Mr. Chrislieb covered about everything. There are 21 tables just in section five of the thesis paper. However, while the research might be thorough, it is my opinion that the experiment had entirely too few participants. This is my greatest concern and complaint.
Twelve is the number of participants that took part in the ROADIE project. For a project that could potentially affect every single connected item that modern society uses on a regular basis (and certainly for thesis level work), twelve is simply not enough. How can proper data be acquired from twelve people? It cannot. I was stunned to read that the experiment only had twelve participants. The author even noted that “due the small sample size it is impossible to find a statistical significance of any of these results” (Chrislieb 2005;60). It seems to me that if the author needs to admit that the testing pool was too small, then there is a serious problem. Ten points were deducted for this shortcoming.
To me, the potential of this project was its saving grace. It seemed that from the beginning of the thesis paper, the project was geared more toward the future. As an owner of several computers, televisions, stereos, home theatre personal computers (HTPCs), two video game consoles, tablets, smart phones, and various other appliances, I constantly agonize over the location of remotes. I am always attempting to find unified solutions to access all of my devices. I install various remote software onto my tablets and phone all the time. The idea of being able to control them all with a common sense language protocol really piques my interest; even more fascinating is the concept of using my actual words to control many of the operations in my house.
The author paints a lovely picture of a futuristic home that is both smarter and safer. The author imagines stoves that alert the user of the presence of a small child to ensure that burns do not occur and microwaves that ready themselves for defrosting when the freezer is opened. This sounds like a world I want to live in. Not only that, but it sounds attainable. It is easy to grasp the crude ROADIE design and throw it forward in the mind fifty years to a sleek, more inclusive interface. The potential for smarter, more intuitive and adaptive appliances is the future.
The author also crafted a section concerning how ROADIE can be expanded. This section is fairly well written in that it not only includes hypothetical situations, but exactly how improvements can be made in the future. For the potential of this project–hypothetically of course since it is a closed project–I awarded 25 points.
Conclusion and Overall Score
The project I chose to write about was one that piqued my interest in the title alone. As a student of the world and someone that constantly yearns to be more connected with technology, I always feel the strong desire to control what I can around me in a unified fashion. This thesis paper illustrates a step in that direction. It presents us with an idea for a device that uses actual human words to consider, command, predict, and execute actions on our various devices. While much of the project is based around simulations and hypothetical situations, it reaches out to a seemingly near future with its designs.
I graded the overall technology, applicability, status, and potential of the ROADIE project. In the end, it received a score of 70. It passes, but barely. The technology is good, and it utilizes what is freely available and volunteer-driven. However, it comes with a substantial number of bugs because of the open source nature of where it pulls its data from. It is presented in such a way that it is only applicable to musicians. I feel that the lack of Internet control was almost criminally flawed. The Internet plays a huge part in our daily lives. The status of the project is closed, since this was just a thesis paper. However, the potential for this idea is huge. The ROADIE project itself seemed geared less toward the present day so much as toward the future. I would definitely be interested in seeing this concept developed more thoughtfully and completely. However, by all accounts, that time is not upon us–yet.
Chrislieb, J. H. E. (2005). Reducing complexity of consumer electronics interfaces using commonsense reasoning. (Master’s thesis, MIT)Retrieved from http://agents.media.mit.edu/projects/consumerelectronics/espinosa_thesis.pdf