"We had years of policy research sitting in PDFs across multiple languages. The challenge was not just digitising them, but making the knowledge inside them discoverable and connected."
The Africa Policy Research Institute (APRI) is an independent, nonpartisan think tank with offices in Berlin and Abuja. Over the years, APRI has produced and collected a large body of policy research spanning governance, trade, climate, security, and development across the African continent. Much of this knowledge existed as unstructured text in PDF and Word documents, often in multiple languages, making it difficult to search, cross-reference, or surface connections between related policy positions.
The challenge
APRI’s research library had grown to the point where finding a specific policy statement, or understanding how different documents related to one another, meant manually reading through individual files. There was no way to search across the full collection by topic, entity, or policy position. Documents in French, Portuguese, or other languages added another layer of friction. A researcher working in English would simply never encounter relevant material published in another language.
The organisation needed a system that could process documents at scale: extract the text regardless of format, translate non-English content, identify the entities and policy statements within each document, and make all of it searchable through a single interface. Critically, it also needed to capture how concepts relate to one another, not just keywords, but the underlying structure of who said what about which policy, and how those positions connect across documents.
What we built
We designed and delivered an end-to-end document processing pipeline that takes raw policy documents and transforms them into structured, searchable knowledge. A researcher can now upload a batch of documents in any supported language and receive extracted policy statements, identified entities, and a knowledge graph that maps the relationships between them, all through a single web interface.
Document ingestion and text extraction
The pipeline begins when documents are uploaded through a web portal. Each file is stored securely in the cloud, and its text content is extracted automatically regardless of format. The system handles PDFs, Word documents, and scanned files, producing clean, structured text ready for downstream processing.
Multilingual translation
Policy documents arrive in a wide range of languages. The system automatically detects the source language using a sampling technique that focuses on the body of the document rather than headers or metadata, improving accuracy for documents with mixed-language content. Non-English documents are translated in full, preserving meaning through intelligent chunking that maintains context across section boundaries. English documents pass through without unnecessary processing.
Entity extraction and knowledge graph
Once translated, documents are processed through an AI layer that identifies entities (organisations, people, regions, policy areas) and extracts the relationships between them. The system builds a knowledge graph that connects policy statements to the actors and topics they reference, enabling researchers to trace how a particular issue is discussed across multiple documents and time periods. Vector embeddings make the entire collection searchable by meaning, not just keywords.
Cost-optimised infrastructure
Processing large document collections requires significant compute and database resources, but only during active processing. The system provisions cloud database clusters on demand and tears them down automatically when processing completes, reducing infrastructure costs by 80-90% compared to always-on deployments. Between processing sessions, the cost is effectively zero.
The result
The pipeline turned APRI’s document archive from a static collection of files into an interconnected knowledge base. Researchers can now search across the full body of policy research by topic, entity, or policy position, regardless of what language the original document was written in. Connections between documents that would have taken days of manual reading to uncover are surfaced automatically through the knowledge graph.
The system processes documents in batches of up to eight concurrently, and the on-demand infrastructure model means APRI only pays for compute when actively processing new material. The entire pipeline is orchestrated through a single startup command, with all five services coordinating automatically, with no manual intervention required between upload and results.