Not Getting Any Research from Your Office of Institutional Research?
Dr. Susan Coia Gailey
Independent Consultant
Is no research forthcoming from your Office of Institutional Research? Had you [and everyone else] envisioned that a central Office of Institutional Research would be the ultimate solution to all of your reporting needs? What you need is a Strategic Planning & Research Office in the President?s Office, and you need to keep it separate from your reporting operation. If you do not agree, do not understand or are intrigued or desperate, read on. If you do not have an IR Office but are thinking about getting one, read on to avoid disappointment.
Offices of Institutional Research often have responsibility for completing external surveys; fulfilling data requirements for your various accreditations; conducting internal surveys; publishing a Fact Book in hardcopy and softcopy (i.e., managing a website); providing figures upon request; providing reports, ad hoc and standardized, and usually proscribed by others. As such, the IR Office is an operation like any other in your university ? a numbers crunching/reporting department, as opposed to a strategic planning and research center. Realistically, your ?office? may consist of only two people plus, perhaps, a student worker who requires supervision by someone who assigns odd tasks. In order to function so leanly, it is imperative that the two professionals have the qualifications and credentials to pull their weight.
The data element is the basic building block for information, which is the basic building block for a body of knowledge. Specifically, data are analyzed to produce reports that give information that, when integrated, yield knowledge. You want to integrate your evolving body of knowledge about your university with information on the changing landscape of higher education because it is the context within which you operate. (Remember the proverbial, ?knowledge is power??) So, let us begin with your data, from which information and knowledge are possible ? ?possible? because your IR Office/reporting operation consumes much time and expends much effort to ensure that information is not built on a house of cards. You will come to understand that your IR Office is way too busy to advance your body of knowledge, and that you need a separate, dedicated research office for research-based strategic planning.
To function productively, one person is often the data manager/analyst, dealing with that fundamental building block ? the data element. Data management is a dedicated function that leaves little time and focus for anything else, much less planning and conducting research investigations. A person who is oftentimes the ?director? depends on the data manager/analyst to investigate data definitions for each data element utilized in reporting. Every data element has a story. Data definition pertains to the methodology by which another department collected and entered the data element at a point in time and associated codes. Errors in coding are common; fluctuating methodologies for data collecting and reporting are common.
Upon discovery, coding errors must be corrected by the data manager prior to analysis. Missing data are common, so he must determine why and whether a given data element has cases for which a data element is systematically missing. The data manager must maintain constant vigilance over each data element because methodologies are subject to change from one year to the next.
Data elements come and go. Departments throughout your university collect and enter the data, often as a byproduct of their respective operations; it is up to the data manager to collaborate with your computer system?s power users and with IT. These people are busy conducting their own respective operations; as survivors, they mainly care about data that enable them to run their departments. Ideally, your reporting operation is partly decentralized, with your power users reporting off their own data from their respective departments.
Because particular data elements are needed for reports, often the data manager should synthesize data sets for this reporting function that he has ?cleaned.? That is, he has checked each data element to determine whether there is missing data, and why. He has corrected bad codes after determining what they should have been. He has captured the data at a point in time at which each data element represents the information that he needs it to convey. With the help of software packages, he saves syntax for commonly needed data transformations (e.g., SATV + SATM = SAT score) and code corrections. It makes little sense to ?reclean? data for every report. Of course, someone inevitably requests a report that involves at least one new data element. Every single time your data manager/analyst receives a data set off your system, he must check it out (i.e., ?clean? it). Therefore, having power users forward data files to him for analysis is of limited help in lightening his work load.
Your data manager also determines how best to organize IR data files for completing prospective reports. For example, sets of UCLA reports might be organized by admission year; these data files multiply quickly, and they are longitudinal (i.e., linked) if you are a full participant. It is imperative to devote time to data management in order to operate and produce valid (correct) and reliable (consistent) reports. As such, the data manager also should serve as data analyst. Reporting off the data files ensures that the data files are, indeed, usable. Done well, data management is a critical, time-consuming, behind-the-scenes function, underappreciated for the time and effort expended by the data manager and by other people with whom he collaborates. Ideally, there is another analyst to help with the inevitable backlog, but this person does not manage the data. As the saying goes, ?Too many cooks spoil the brew,? not to mention that data management is so time-consuming and requires so much focus that it must be a dedicated function to produce valid and reliable reports/information.
Decentralized reporting makes sense because: (a) power users are experts in their data; (b) reporting off their data encourages data integrity (e.g., coding errors are corrected on the system because now they are the power user?s headache) and (c) the workload is dispersed to improve work flow. Department heads will disapprove of this arrangement because outside requests detract from internal (intradepartmental) requests. That is one major reason why department heads were in such strong support of a central IR Office ? to free up their own personnel.
As a central repository for requests, the data manager must be in production mode, with little time for consultation, because of inevitable and unrelenting backlogs. To expedite production, the IR Office/reporting function might issue a form for submitting requests, which might require that you name row and column headings for the spreadsheet that will be prepared for you. Most people outside IR do not have the knowledge to be so specific, nor should this be expected, so requests for reports ? as single figures or more complex tables ? are often half-baked. Methodologies for producing reports must be aligned with their intended use for the situation at hand. Questions arise such as, ?On what population shall we report? Shall we count graduates as persisters [which is appropriate when measuring institutional effectiveness] or gone [which is appropriate when measuring revenue].? It hampers IR to clarify requests, and it often is a challenge to do so. When requests are made second-hand (are you guilty of this?) your appointed messenger cannot answer questions. A best-guess by your IR Office might approximate the ?right? figure(s), or be downright misleading. Misinformation is worse than no information.
Oftentimes reports that are provided as requested are not what people really needed, so the recipients toss them aside. As such, IR reporting is often a waste of time. Your IR Office is the resource for producing wasted reports.
Adding to the IR backlog are requests for information that would be interesting to know, rather than useful. It?s so much quicker to ask [for a report] than to fulfill [the request]. Your IR Office is the resource for the production of reports with limited value.
Reports generate more questions, so recipients are back in line ? except for people with clout, in which case other requests, which could have a potentially greater impact on your university, are bumped back.
Adding to the backlog of requests for reports is the time-is-of-the-essence external reporting to the government, and for publications and accreditations. Whether reports are required by the government, deemed essential for visibility in the marketplace, or for accreditations, figures must be collected and validated and cross-checked against previously reported figures and figures that appear in other publications. Oftentimes each inconsistency takes several hours to reconcile.
Some IR Offices/reporting operations coordinate the distribution [of sections of external reports] to power users of the departments in which the data originate, rather than retrieve data from the system or from Office cleaned and maintained data files. Nevertheless, this coordinating function by IR requires their follow-through to ensure that deadlines are met, and this decentralized system of providing figures does not circumvent centralized review and reconciliation by IR.
In the case of internal surveys, your IR Office is often responsible for administration, data collection, analysis, and reporting. Due to the confidentiality option to students, reporting of UCLA surveys requires data transmission to UCLA when you request a report that necessitates merging survey data and university data. UCLA also has a backlog of report requests, so monitoring requests is another administrative task under the purview of your IR Office.
Your IR Office often contends with analysis on data from illinformed, internally developed surveys. For example, a common inquiry is ?Let?s find out why our students are leaving,? which precipitates a survey of drops and descriptive statistics. Your IR Office might have no input: You need a comparison group (i.e., persisters). Self-reports of drops might be similar to self-reports of persisters, had the latter also been surveyed, i.e., statistics might not differentiate drops and persisters. Your IR Office is the resource for the production of meaningless reports.
Additionally, your Office might be sent isolated data files for which there are no cogent research questions; oftentimes these data files arrive without accompanying data definition and population descriptors. These require follow-through for clarification, usually by the director who is appreciative when people are receptive to being questioned by him. He is appreciative because people tend to want their reports and wonder what?s so difficult about that. Moreover, they want reports, not advice. In order to build credibility over time, each IR staff must have the right qualifications, as opposed to being one more person with an opinion. Universities have no shortage of people with opinions.
The Fact Book, in itself, really does not advance the institution in any discernable way, but it, too, is often the responsibility of the IR Office (i.e., two qualified professionals). Your IR Office gains consensus on content and methodologies to suit review by your various constituents; reconciling opinions can be a grueling process. It is responsible for completion and verification, with online and hardcopy publishing entailing different sets of procedures.
Is it any wonder that there is no time to consult with university leadership and to conduct research investigations to inform strategic direction, strategy, policy, and executive decisions?
The IR director fills the role of consultant and monitor/gatekeeper, though your IR director also must participate in generating reports. Therefore, your IR director must have the same capabilities as the data manager/analyst, but he keeps data management as a dedicated function to enable appropriate focus. The data manager provides the director with data subsets from which the director also can produce reports. In spite of his role as gatekeeper and organizer of the workload, people just want their reports, so full cooperation from IR staff is politically correct.
The director, data manager and data analyst must be multiskilled in order to pull their weight and justify your IR Office budget. In addition to managing data and reporting, the data manager/analyst often is the one who administers internal surveys and completes external surveys. However, each person must be able to grasp the logic of ?methodology? to critically review his own work. In order to enable the data manager to focus on the arduous details of data management, the director double-checks (as opposed to corrects) reports by the data manager/analyst prior to send-off, prioritizes and clarifies incoming requests, and completes reports; he strives to ensure appropriate interpretation and use of reports when the recipients agree to his input, though your IR Office is pressed to increase report production.
Should you add another person to this reporting operation, it would be another data analyst to pick up more of the slack in reporting, bearing in mind that demand will increase with production, but the IR director might be able to give more attention to fielding requests to ensure appropriate methodology. It makes little sense to hire additional people only with secondary skill sets, such as performing internet searches, completing external surveys, administering internal surveys, or developing items on internal surveys. These are secondary skills. They are possessed by your IR director and data manager/ analyst. They are not stand- lone skills. Without full skill sets your IR Office/reporting operation is people-rich and skill-poor ? the IR Office contains fluff.
A separate research office in the President?s Office would plan and conduct exploratory research investigations, usually ex post facto, which utilize your historic data, and integrate quantitative and qualitative data about your university and higher education. This research office conducts confirmatory research to update and refine findings and note trends. It can depend on the reporting office (the data manager in your IR Office) for data sets. This ?office? might consist of just one highly qualified person, with administrative service available in the President?s Office. The director collaborates with leadership throughout your university.
Should you merge these two functions ? reporting (IR Office) and research (Strategic Planning & Research Office) ? under one director, the reporting operation will consume combined resources. Your real research office focuses on strategic direction and strategy: Planning and conducting enrollment management investigations, which cover the admissions, financial aid, and academic aspects of your university, with learning/performance and graduation being the usual outcomes of interest.
Bear in mind that predictive modeling is statistical analysis, which is part of research, but not its essence. Sometimes consultants are called upon to perform predictive modeling and simulations. They do not necessarily conduct exploratory research investigations. The similarly trained director of this planning office should have the best qualifications to be the primary contact for such consultants in order to capitalize on their service. Generally speaking, before hiring consultants in an advisory capacity, consider that your own people know more than you think; the knowledge you are seeking might not be so exclusive and rare that you need outside mentorship. Even the most gifted consultant would be more valuable with years of experience at your college as opposed to being a visitor. Did you learn everything noteworthy about your own university in a matter of even several months?
Your directors of each office ? reporting (IR) and research (Strategic Planning & Research) ? should have a doctorate in a behavioral science and should have conducted empirical behavioral research in the doctoral curriculum. Specifically, they planned and executed thorough literature reviews to develop a theory- and research-based hypothesis to direct their research. Additionally, your Director of Strategic Planning & Research has evidenced an aptitude for comprehending the dynamics of organizational systems. The data manager/analyst might have a bachelor or masters degree in a quantitative field in which data management and data analysis were inherent. All professionals should have benefitted fully from their educations with excellent critical and analytical thinking skills. A strong undergraduate liberal arts education is essential as a basis for acquiring new information, and to fathom the unfamiliar throughout life and career. Some Master?s educatedpeople fall short of this caliber.
Remember that knowledge of a software package such as SPSS is necessary, but not sufficient. SPSS is a software package like any other software package. One can run a program without producing an error message, but doing so does not guarantee valid and insightful results. After all, does knowledge of Excel qualify its user to prepare financial documents? Does knowledge of Microsoft Word guarantee a best selling novel? The Director of the Strategic Planning & Research Office initiates, plans and conducts exploratory and confirmatory research investigations to provide strategic direction and evidence-based options for strategy and policy to ensure a thriving university. He collaborates with your similarly educated IR Director to identify patterns in IR reporting requests that could disclose unmet research needs. Like other departments, he is served by the IR Office, but the data manager/analyst also has a dotted line to him. The Director of Strategic Planning & Research consults with leadership over topics of executive interest and issues, possessing a good degree of intuition and insight into their concerns, and he investigates accordingly. Anticipating future needs, he also is highly pre-emptive. He offers information and conveys knowledge that is difficult for you to hear, but that you need to know; you discuss implications and what you need to do about it to ensure a healthy future for your university. Your directives cannot be expected to be so specific so as to name each analysis that would produce the ?aha? ? the ?aha?s? will be forthcoming with a talented Director. The qualified Director must be free to exercise rofessional judgment and discretion, and not be micro managed, because a knowledgeable Director knows where he will hit dead-ends. As such, he should report to the ultimate decision-maker, usually the university president, without a layer of management between them, while your Director of your reporting function (IR Director) might report to your Executive Vice President, who may oversee university operations. After all, how many people do you need in senior positions that make sure other people are working? Ask, ?What is the technical know-how that justifies the senior position?? Review your allocation of resources to improve productivity and university development. Doing so would improve upon the productivity of all of your functions.