Taking stock of the state of in-house lightweight data analysis

Nonprofit projects are adopting software and data systems for monitoring and tracking purposes. These are done through pre-developed dashboards and reports. But in this article, we look at whether organizations are doing data ad-hoc data analysis to gain insights about their work, what determines the success, and inherent challenges of nonprofits in analyzing the data in-house.

In-house data analysis in nonprofits is traditionally been done by using tools like SPSS, Excel, Stata, and R. There are trained public sector professionals who specialize in these tools. This approach fits projects that require research. Usually, this is done by someone who is not running the program but by a dedicated personnel/team. Let's call it heavyweight in-house data analysis and very few projects are required to do this.

But is there is a much wider need for in-house data analysis which can be done in few hours than few weeks?
1. People who run programs have questions and curiosity about their projects. Since they also have data, they want to query the data to get some rough answers to their questions. This is extremely important for the continuous improvement of the program.
2. When a project is completed, the software provider hands over dashboards and reports. Before the data is in place and the project is in full swing - it is very difficult to pin these down exactly. But when the data is in place (maybe due to lack of funds) the funds for software are exhausted.

Even when the best of the analytics and data warehousing tools are available, user-driven data analysis has a learning curve. Without the conceptual/intuitive understanding of entities/tables, relationships between them, grouping methods, drilling along dimensions, etc - there is a limit to how much one can do. Many tools allow the above using visual methods, but one still will have to tell it what it should do. The tool cannot do that by itself. Hence in my assessment, there is not that much scope for tools to help you further.
On the other hand, budgets are constrained hence paying a software provider to help with reports, dashboards on an ongoing basis is an expensive proposition.

Some organizations that have long-running programs have been able to learn and get better over time with data analysis. Shorter programs that run only for few months, do not provide the opportunity - hence unless one already has someone in-house who can do it, it doesn't materialize - given other responsibilities.
Secondly, what follows is that there should be some funds available (or not used upfront) so that the right type of dashboard, reports can be built at the right time.

Finally, economically speaking, one can argue that the value of such in-house ad-hoc analysis has not been recognized yet fully, or maybe the value isn't there (at least relatively speaking compared to other priorities). But if the value is there and once it is recognized an economic solution will emerge. Organizations will have in-house people to do the analysis, or have formal training in data analysis for their program people, or will have funds to hire technical support for such work. To a large extent, it is not a technology problem.