DOCUMENT RESUME EA 002 382 ED 030 967 By -Case, C. Mar stork The Application of PERT to Large-Scale Educational Research and Evaluation Studies. Educational Testing Service, Princeton, N J Report No- RM -69- 14 Pub Date 8 Feb 69 Note 9 16p, Paper presented at the Annual Meeting of the Amer. Educ. Res. Assn. (Los Angeles, Calif., Feb. 8, EDRS Price MF-S0.25 HC-S0.90 Descriptors -* Critical Path Method, Educational Researchers, Manpower Utilization, Networks, *Ob|ectives, Program Coordination * Program Evaluation, * Program Planning * Research Projects, Resource Allocations^ Task Analysis, Work Simplification Identifiers -PERT, * Program Evaluation and Review Technique The application of a PERT system to a large-scale pro|ect will increase the probability of accomplishing project objectives by providing greater visibility of (1) the project objectives, (2) the relationships among the parts of the projects, and (3) the relationships of those component parts to the project objectives. Consequently, educational researchers should find that a PERT system will increase their collective ability to coordinate planning and evaluation efforts. A review of a PERT system applied to a study of disadvantaged school children indicates that in order to achieve optimal utility, the system should be implemented well ahead of the project starting time, and must be responsive to program developments through continued updating procedures. (JH) o rm-69-1^ [N* vO O l!*\ o Q LU RESEARCH MEMORANDUM THE APPLICATION OF PERT TO LARGE-SCALE EDUCATIONAL RESEARCH AND EVALUATION STUDIES C. Marston Case U.S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE OFFICE OF EDUCATION THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROri THE PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPINIONS STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION POSITION OR POLICY. Paper presented at the meetings of the American Educational Research Association, Los Angeles, California, February 8, 19^9 • 04 00 CO Educational Testing Service 04 Princeton, New Jersey o o May 1969 | E r ERIC THE APPLICATION OF PERT TO LARGE-SCALE EDUCATIONAL RESEARCH AND EVALUATION STUDIES This morning I would like to cover four points related to the manage- ment of large scale studies. First, I will discuss the need for what I call "project visibility." Second, I will note some basic features of a PERT chart using the example in the handout. As some of you already know, PERT is an acronym standing for Project Evaluation and Review Technique. Third, I will use a very simple application as a basis for making some recommendations about implementing PERT. Finally, I will describe an application of PERT to a large scale project at Educational Testing Service. In this last item I will refer to the reports in the handout, which come from one of IBM’s PERT- like computer packages. The success of any project depends on whether the participants actu- ally contribute to its accomplishment. This may seem obvious, but one often finds that the participants in a large-scale project, including the manager, are vague about how their efforts are related to the project objectives. Such projects often fail to meet their objectives; sometimes the objectives have to be changed so that they conform with what the people on the project have been doing. But we try to avoid that. Usually when one finds that the efforts of the people working on the project are not well related to the project objectives one will also find there is a lack of project visibility. I feel that this visibility can easily determine the difference between a highly successful project and a not so successful or an abandoned one. Three things need to be made visible: (l) the project objectives them- selves, (2) the relationships among the parts of the project, and (3) the relationships of those component parts to the project objectives. - 1 - - 2 - Figure 1 in your handout is an example of how we can go about improving project visibility. It looks rather like a flow chart, but strictly speak- ing it is not one. It is a work item network. Each box represents an item of work. The leader of a certain part of a project estimated that this set of 23 work items was necessary to the completion of his part of the project. In a work item network, a key relationship among the work items is made visible. This key relationship is precedence. The arrows in the network mean that the work item following the arrow cannot begin until the work item preceding the arrow is finished. Thus one sees from a work item network how the work items are related to each other in terms of their succession in time. Our objective is the completion of all of the work items, but since the prece- dence relations all flow into the last work item, we may as well say that our objective is the completion of that last work item. Thus far we have defined precedence relations among the work items . Nov let us make this network into a resource allocation scheme where the resource to be allocated is time. We can do that by assigning time durations to the work items. The number at the lower right corner of each work item box is the expected number of days needed to do the given work item. Now let us see how PERT fits into the picture. We have in the network diagram of Figure 1 all the information contained in any PERT chart . 1. It has work items (the boxes). 2. It shows the precedence relations among the work items (the lines and arrows ) . 3. It shows expected time durations of work items (the numbers in the lower right corners). ERJC - 3 - PERT is a method of shoving proposed resource allocations; in using it one develops a time allocation scheme, and perhaps a scheme for allocating other resources, such as cost or manpower. It has been successfully employed in managing large-scale projects in the construction industry and in the Depart- ment of Defense. It was iirst used in the development of the Navy's Polaris missile. Nov a full-blown PERT treatment of a project involves the use of a com- puter to analyze the network before the project starts, and then periodically during the execution of the project. In small projects, however, worthwhile benefits can be gained with the use of a PERT-like diagram, without getting into more sophisticated analyses. I will give you an example. Recently my wife agreed to the use of a PERT chart for cleaning the house and getting ready for guests later in the day . We drafted a network diagram which gave a clear view of the day's tasks and the precedence relations among them. Rooms had to be put in order before they were dusted and then dusted before they were vacuumed. Operations in the various rooms could run along roughly parallel except in our little boy's room, which was done last, so he could play there while we were busy elsewhere. The dishes could not be washed until all the cooking was done. The roast was to go in the oven at a certain time, and so on. The chart worked quite well. It not only kept us at relevant tasks in their proper order, but it became an unobtrusive rallying point; we worked in reference to the chart, with no need to ask or tell the other what to do. Now, this unobtrusiveness is an important point in coordinating research projects, because most people, especially researchers, do not like to be told what they are to do, in any direct way. This is one reason I think -k- that educational researchers will find that a PEET chart makes it easier to coordinate efforts with others on large-scale projects. We have been noticing the fact that the PERT chart, a detailed and explicit document, is a help in maintaining momentum toward project goals and that project morale is benefited by such momentum. However, the rERT chart can be a product of management alone, with little representation of the desires of those who are to follow its dictates. On the other hand, a PERT chart can be the product of the joint efforts of project management and the other participants. If it is a joint product there will be a spirit of coordination and cooperation; done in this spirit, PERT helps to avoid that obtrusiveness sometimes ascribed to management. Wow, people are usually able to clean house without the benefit of a PERT chart. However, these sorts of simple applications of PERT are invalu- able in giving one a sense of how PERT works. I recommend that prospective PERT users make a few such simple applications before they attempt the use of PERT in a large-scale project. A PERT system provides a method to make plans before a project begins and then to review and revise the plans while the project is being carried out. In the house cleaning, for example, we jointly prepared the PERT chart and then referred to it when each work item was done in order to choose the next thing to do. During the process of doing these tasks we noted down the times each work item was started and finished and compared them with our planning estimates to see whether we would have to work harder or would have time to spare before the guests arrived. This reviewing and revising while the project is going on is crucial in keeping the project on schedule, largely because it keeps the relationship of -5- everyone's efforts to project objectives visible to all concerned. This process, known as updating, should be a regularly-scheduled event during the lifetime of the project. I think updating is the most demanding part of keeping a large PERT system going, because the people whose work is represented on the charts often have to be reminded of the need for infor- mation about their progress. You can use various devices to prod the par- ticipants for this information; you can set up a reporting schedule and provide progress report forms; you can write memos, make phone calls, and visit the project participants. Such special efforts are frustrating!^ inefficient but it will not be necessary to resort to them so often if the project is well planned early enough and with all the people in the project participating in the planning. Now let us turn to an example of the application of PERT to a large ppoj ect . A group of people at ETS have embarked on a large-scale longi- tudinal study of disadvantaged preschool children. Another person and I were attached to the project as advisors and implementers of a PERT approach to the coordinating of the project. We came into this role more because of our ability to bring a computer to bear on the problem than because of management skill or experience. We have used an IBM PERT— type computer package called Project Control System, or PCS, which runs on the IBM 360 computers . At the outset, the project had seven component parts called task forces, with a task force leader designated for each. Each task force was to provide a way to assess a different aspect of the preschool child or his environment. The main objective of the project is to identify the components of early - 6 - education in preschoo] and primary grades that are associated vith children's cognitive, perceptual and personal-social development. We first worked with the task force leaders in setting up work item networks. Figure 1 is one of the early task force work item networks. In the initial seven task force networks there were about 220 work items. About a month was required to gather this ?Ln format ion and get it into the computer. Once we had this initial version in the computer, we were able to pro- vide schedules and graphs corresponding to each task force's proposed plans. Page 2 of your handout shows an example of a schedule report corresponding to the network in Figure 1. This tabulates the work items and the earliest and latest they can start and finish. It also indicates the float or slack associated with such scheduling; this is the amount of leeway one has in doing the given work item providing everything else goes according to schedule. Notice that in some cases this is a negative number, which indi- cates inconsistencies in the proposed plan. Page 3 shows a graph of some of the same information. We do not need to examine all the details of these reports here although you may want to study them later. In this project we are estimating manpower requirements for each work item. We are doing this in order to avoid excessive manpower demands in peak demand periods which would be costly or impossible to meet . The IBM program prepares reports showing manpower requirements versus time. Pages h and 5 of the handout are examples of these. Page h , for instance, shows a tabulation of manpower requirements of various work items by days, with the total requirement of the project for each day at the bottom. On page 5 the resource utilization report shows, with asterisks, the total require- ment for each week and, with C's, the cumulative requirements of the project over the term of the entire project. - 7 - There are three problems we have encountered in this implementation of PERT which seem to be of sufficient importance to mention here. First, we did not have the initial version of the project (the planning version) on the computer and ready to go at the time the project started. This meant that people were working by plans, some of which were later shown to be inconsistent. That is, they could not use as much time as they were alloting themselves and still meet certain deadlines. This is a perennial problem: the problem of gaining time enough before the project starts to prepare a plan which is at once consistent and acceptable to all concerned. The second problem we have encountered to some extent derives from the first; at least it is aggravated by the presence of the first. This is the problem I mentioned earlier, keeping the PERT system up to date. As soon as the project is started, updating should begin. This means that periodically, say every week or two, the network should be reviewed and revised estimates of resource requirements should be provided. If any work items have been started or finished since the last updating, they should be reported. When update information is put into the computer system, work status reports such as the one shown on page 6 of the handout can be provided. This report tabulates information on the work items which have not been completed. It includes the original estimate of time required and the present estimate of remaining time needed to finish the work item, plus other information. In our project updating has been somewhat thwarted by the fact that other things received priority over the updating activities. The third problem, which is not independent of the first two, is that the PERT materials are not sufficiently used as a basis for reviewing - 8 - and revising project plans. Now this is the one key use for which PERT was designed. The main reason PERT has not been used in reviewing and revising plans (aside from the problems of a late start and weak updating) is that the project staff has had to contend with a variety of unexpected events. These repaired major revisions in project plans, such as personnel changes and changes in the subjects to be studied. As a result the staff has often, in critical times, fallen back on ad hoc measures to revise project plans. In this project we are continuing to work on bringing the status of the plans up to date. We expect the PERT system to be of more benefit as the project progresses, plans crystalize, and the PERT system catches up with the plans . I believe that a PERT system, implemented well ahead of project start and kept responsive to project developments, provides a certain visibility of the structure of the project not available by other means. Several benefits derive from such visibility. Basically these stem from the fact that everyone concerned sees how the pieces of the project fit together in support of the project objectives. With this readily available grand view of the project, ramifications of any changes are quickly assessed and massive changes are more easily handled. The visibility aids in maintaining communication up and down and across project organizational lines, with a PERT network acting as an impersonal entity to represent the wills of all project participants. My conclusion is that such visibility, in an other- wise well-conceived project, will make it more likely that the project objectives will be accomplished. THE APPLICATION OF PERT TO LARGE-SCALE EDUCATIONAL RESEARCH AND EVALUATION STUDIES C. Marston Case, Educational Testing Service Part of the Symposium: Management Problems in Conducting Large-Scale Research and Evaluation Studies 1969 American Educational Research Association Meetings February 8, 1969, Los Angeles PROJECT BASE OATE t JUL 68 PROJECT DURATION 667.0 PROJECT COMPLETION OATE 10 OCT 69 i r\ Z UJ UJ 00 z © ac UJ UJ © a UJ to to I H- 15 15 12 ; © ! Or t ui I l a o o ►- c z 1-4 3 O' O UJ T a. < o a x a k 1-4 DC ac < ULJ H a to 44 tn 55 kk *4 M & a 33333. •4 -* * 1 ! *4 -» < M • Iff ft# » m! •4N «M • «< 0 fl 0 o 0 flO!attiO>* 9 >d»» ~ e 0 00 9 « 4 <|R>e 0 4 l 4 N 4 i nO> 0 :i 4 ^ 4 l 9 > • • ♦ •••'•**•• # #«*••**• tit • ooooNN^^f o^iAO^me^i ttnt 0 4 ’ONKacO (^OOf 4 « 4 Ni w © * * 6 * 6 « o UJ (0 ) i »h i | MM I I I ftO o ■ t* in fO •* O O * * o er> 22 30 I I a I i I * I ii ; I* i I I ! I I • o © to m rv • o o 4 * o fNJ • o © m m S I • o o CM O • o o m in #;##*>•***# *6 *66 * 6 * 6 m 6 I * o ou u o u o u o o o o * • # u u * o o ♦ * * * * # * « * * # * * « * # * a a « # « « * # * * # « UJ UJ -J «j < < o o 00 00 O UJ C > M M ac o* uj < a. .j © a • •••••••••••••§•§•§••••••••••••••••••• ooco^JCT'ococcpofr. iTN-omirmir. ♦ OOOOOOCOaDOOOOCOCCCOCOOCOOOOQOCOCOXcOXOOCOaOCOOOCO^a'OC'O'a'C'a'CrC'a'C' ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ o ^ ^ ^ >n e e o e e e o o %© ^ *£ ^ ^ ^ ^ ^ ^ ^ jcoooaQ.&a.ah^hk>>>>uuuuazzzzecoeccDtfoc 2 Z Z ZZZUJUJUJUJUJOUOOOOOOUJUJUJUJUJ<< 4 C < 3 < $ n * o x o x u. x Ui k to > to o 00 k Z o o o Ui -3 c X a z ce * * co i/d Ui O t/i 3 i/> >0 x o 3: 0 1 oo 00 tn o 3 k 1A k V) o < UI X co k UI UI -J k 0 C o 3 k Ul z CO 0 CD -H 3 in Ui 5 o z 3 a o oc a Ui u z Ui 5 IS ui 2 CO UJ z § 3 UJ ae *y) I I SI H < I CO Q I UJ | kX I < CO I J 2 I CO <0 cd 3 < N t H UJ Qk UJ «£ k- a u Ui k a. OC K< Ui k t/D 1 a i 1 pH H pH p pH pH pH pH p* ft. «u < i i i i f * IT CD ID ID in in in in •n ♦ k I I ! > V > k k k >- k > V pH 1 ** 1 1 < < < < < < < 4 t < < o o 3 1 Z 1 1 3 1 1 o 1 o e O O o o o o c k 1 | z pp 1 Jk | 1 o o o C o o o o o O 1 CD < 1 1 • • 0 0 • 0 0 0 0 0 < z 1 « OC 1 t c o o M o in o o in o o 1 X 31 i rt> pH m pH fVJ rvj pH pH <_! 1 O Q 1 1 k k | 5a | OCX I uj a t a. a i i GO <0 3 -3 O o o CO 00 CO CO C0 00 00 3 00 HD H> <0 HD H> HD HD > 52 > k- a k > > U CD 3 O U Ui a o a UI Z < Z a to 5 z z Q HD O k a PM 3 k c* *n m pH pH pH pH CO <30 00 00 00 CO 00 3 3 HD HD 0 PD HD HD 0 HD HD CO CD a Cl a ft. ki > O 3 3 UI cu LU UI a O UJ < PC CO t/D to CO o z Q k CD pn pn m * nj o pH N pH eg IM pH PH « k < < X oc s§ -J < I 2 HD o o o CD o c O o o 0 0 0 0 • 0 0 0 0 0 O o o o o O o o o o o o o o o o o o o o 0 0 0 0 • 0 0 0 0 0 PM o o PM o in o o m o pH 4- pH m pH PM eg p4 pH j H CO l— 1 CO k- CO 1 UI 3 U 3 UI i k- HD UJ HD k i u> 3 UJ i QC U 3 C X i a 3 3 3 a i CO < i > X Z 1 X UJ _ 1 a CO X C I k* LL> X 1 < < Um 3 p* 1 X to 1 G o k* I -J Ui H? u. z 1 a k O k o a i X < < i UJ O 3 k PH | z X 1 Z c X 1 PH < a 1 X X UJ a i a X i < u. CO 1 m c o 1 UJ z ui t X z < 1 < c a i a fcH k i UI k- UI i X o < eu X i a o z o a i p a CD a X i z PH UJ i < PH CO PH k i •j 3 UI 2 Z i a a o CL pp CO >0 o 3 < CO >0 CD 3 < 00 *0 CO 3 > a o z z z Ui X 3 OC 00 k <0 to Z > f* o z 0 o o X o PVJ > c o k k k 2 O' pH m pn o m k »H k pH k Z PH n. a. O PH UJ k 3 mJ X CO c0 k PH 3 UI mJ CO 3 X k UJ Uj Ui < UJ Ui < Ui Ui k UJ Ui UJ k k CD k PH k > PH k k CO k k O k < < Z -» c O 3 U < CL tO < eg * pH pH a cl 3 2 a ft. a ui oc a Q Z < UJ to -D HD 00 pH O a CL > PH PH Ui 3 2 oc a a < oc CD z oc a to cc 2 & I I I ~! 9 2 1 l i o "D < i/% PM t HD 3 PH m 3 3 O O o O C P0 k k pH pH a X X a * ft. a a ft. & a o ERIC