NIEM Newsletter
Tech Focus: Representing Idiosyncratic Data for NIEM IEPDs
By: Bill Blondeau, WIJIS Data Architect, Wisconsin Office of Justice Assistance
Editor’s Note: NIEM News readers represent a broad audience of decision makers, stakeholders, practitioners, and users. Tech Focus features will cover topics that are more technically oriented for experienced National Information Exchange Model (NIEM) users and programmers. This month’s Tech Focus is about the Wisconsin Justice Information Sharing (WIJIS) Program. This Tech Focus article offers an overview of the issues faced by WIJIS as well as a link to a much more comprehensive article. Special thanks to Bill Blondeau for providing both versions of this article for NIEM News readers.
Real-world situations sometimes yield data that are too irregular to model efficiently via the Unified Modeling Language™ (UML) class diagrams stipulated by the Information Exchange Package Documentation (IEPD) lifecycle process. WIJIS recently encountered such a situation and found an effective and generally applicable solution that is well supported by NIEM—external unconstrained Resource Description Framework (RDF) attachments defined at the Instance level, rather than the Class level.
When You Cannot Completely Model Your Problem Space
The recommended definitional artifact of information exchange in NIEM is the IEPD. The IEPD specification defines both a structure for IEPD representation and a process (the IEPD Lifecycle ) by which IEPDs are conceived, designed, built, and, over time, updated.
The "Information Exchange Mapping and Data Modeling" stage in the IEPD Lifecycle process prescribes a very powerful and mature information representation—the UML Class Diagram. Class diagrams are efficient, expressive, easily understood, and provide good guidance when creating software artifacts to represent static structures, such as information exchange messages.
Class diagrams, however, were never designed to represent information structures that are not completely understood in advance. This can be a nontrivial issue for certain kinds of problem domains. Sometimes the data really are highly idiosyncratic or maybe it is just very difficult, for practical reasons, to assemble a high-confidence sample set of those data. Either way, the practical result is the same—you do not expect your model to hold up in the real world. You expect that new, unprecedented data will come along, in the course of real operations, and break it.
In Wisconsin, the WIJIS program ran into that problem in the course of developing a NIEM-conformant exchange of drug investigation information. This work was funded in part by the Bureau of Justice Assistance, through the National Governors Association's NIEM Policy Academy project. As a spinoff to that project, WIJIS has developed a suggestion for what seems to be a satisfactory approach to this problem—one that requires absolutely no changes to the IEPD definition or process and no changes in the structure of the implementation of the NIEM model. In fact, this approach is only feasible in the first place because the NIEM design team made some very good fundamental decisions.
NIEM Information Exchange Packages (IEPs) are the foundation of NIEM’s business value as an enabler for nationwide information interoperability and sharing. Broad adoption with collaborative sharing of adoption and use experience, best practices, lessons learned, IEP Documentation (IEPD) reuse, and cooperative development IEPDs for nationwide information exchange are critical for achieving the full potential of NIEM. Since the release of NIEM 2.0, there has been a groundswell of IEP development. There are now more than 70 NIEM IEPs registered in the GJXDM/NIEM IEP Documentation (IEPD) Clearinghouse (www.it.ojp.gov/iepd/). In addition, the DHS has more than 50 IEPDs, most of which are in development or operational.
A much more detailed and technical description of some of the technical and procedural characteristics of this proposed solution is available at http://wijiscommons.org/articles/drug_case_rdf/. Three things to note about that article:
- It is currently in peer review and early “Request for Comment” status, which means we can and will repudiate or deny anything later if we want. (It also means, of course, that substantive comments are most welcome.)
- WIJIS has not implemented this scheme at all, and the devil is always in the details. Logically, this should work beautifully. Practical implementation in the real world could be another matter entirely.
- The article describes how we were led to the question of representing irregular data via narcotics intelligence practices. This should not be taken to imply that we found intelligence data inherently difficult to characterize. The particular situation we encountered was simply that the potential problem domain included data maintained in numerous ad-hoc systems, with no standardization of content or structure, and that the requirements for operational secrecy were quite high. Putting together an adequate set of sample data would have been tough and well beyond our resources or those of the investigators with whom we worked.
External RDF Attached to IEPD-Conformant Instances
WIJIS's proposed approach uses the W3C's RDF language, which is highly flexible and easily adapted at any time, to represent unruly instance data. RDF is all about associations between items. In fact, RDF is a graphical, rather than text-based, language; it depicts the data as a diagram of links and items, in which links are associations (called “predicates”) of known types, and items are either unique Resources or literal values:
Because RDF's link types are identified by URI, it is very flexible. If some instance data does not seem to be expressible by any known RDF association definition, it is easy enough to invent a new association type on the fly and use it. In this proposed technique, the RDF is therefore unconstrained by any preconceived restrictions.
The RDF is attached externally to the IEPD data by a convention for assigning URIs to the individual data items in the instance. (If you can assign something a URI, you can write RDF statements about it.) Conceptually, that works something like this:
In this diagram, the RDF items are linked to individual parts of the XML document.
Well, that is the concept. What would an actual serialized RDF document connected to the XML look like? There are a couple of ways of doing it, and discussing them is perhaps overly technical for this article. But they are not hard to do—RDF serialization is a well-understood practice.
One important note—this has nothing whatsoever to do with implementing the NIEM model in RDF. That is a very large (and praiseworthy) effort that would result in the same familiar NIEM 2.0 model but written in RDF instead of the XML-centric DOM. That is all NIEM—just a different way of expressing the data. This current proposal, on the other hand, is much more modest—nothing but a suggested technique for attaching non-NIEM (and non-any-other-data-standard) information to an IEPD-compliant instance message—attaching it to the “outside,” you might say.
Why NIEM Works
The reasons for this are fairly technical—sort of inside baseball for abstract data designers—and are discussed in the referenced article. For present purposes, it is sufficient to say that NIEM:
- was defined as an abstract, rather than an XML, model
- was built to be RDF-compatible from the beginning
Neither of these two characteristics is fundamentally necessary in order to establish the logical integrity of this proposal. However, if NIEM had been defined as, for instance, an XML data model, with no stipulation of RDF compatibility, WIJIS would not have been able to afford to establish that integrity.
And that is really perhaps the most important thing to take away from this article. It may be that our proposal will turn out to be unworkable in practice. It may be that it contains some as-yet-unnoticed lapse in logic that will embarrassingly doom the whole idea, but that does not change the fact that NIEM's beneficial characteristics made the venture cost-effective in the first place and will probably have the same effect for other efforts.
This is not a small thing. Good data model design pays off, and it keeps paying off as you encounter new situations. The NIEM designers really do deserve recognition for getting these particular things right.
For more information and/or questions, please contact Bill Blondeau at Bill.Blondeau@wisconsin.gov.
Statement for the Record of Charles E. Allen: “Information Sharing at the Federal, State, and Local Levels,” July 23, 2008
Excerpt from the testimony of Charles Allen, Under Secretary for Intelligence and Analysis for the Department of Homeland Security, recently delivered to the United States Senate.
DHS also is working with its Federal partners on a number of less visible but still very important efforts to improve information sharing. I want to highlight two notable examples of initiatives that are enhancing our capabilities: National Information Exchange Model (NIEM) and suspicious activity reporting (SAR).
In the last twelve months DHS has dramatically increased its adoption of the National Information Exchange Model (NIEM). NIEM is a data standards management initiative co-sponsored by the Departments of Homeland Security and Justice with extensive participation by State and local stakeholders. The implementation of the NIEM standard in an information technology (IT) system enables data to be translated into a common language and shared more easily with other IT systems. This effort is essential because without data standards, system-to-system data exchanges are often difficult, both within agencies and with external partners. Data standards like NIEM not only enhance our ability to connect the dots that exist in numerous IT systems, they also enhance our ability to categorize data and ensure appropriate user access and usage in accord with privacy and civil liberties rules.
Across the Department, components such as CBP, FEMA, ICE, NPPD, S&T, TSA, USCIS and US-VISIT are realizing these opportunities through NIEM adoption within major IT investment programs. These opportunities, such as those being built at ICE in support of law enforcement information sharing, will improve the way information is shared with State, local and tribal partners. Additional opportunities will include improving screening against the terrorist watchlisting, defining exchanges in support of radiological nuclear detection systems, and the creation of person-centric query capabilities that will enable agencies, such as USCIS, to gather information from DHS and Department of State systems to build a comprehensive picture of an individual to support immigration benefit determination process.
NIEM adoption is also happening at the State and local level. As part of the Homeland Security Grant Program, DHS requires all grantees to use the latest NIEM specifications and guidelines regarding the use of Extensible Markup Language (XML) for all HSGP awards. DOJ has the same requirement for several of its grant programs. Far from resisting this imperative, our State and local partners are embracing the adoption of NIEM because it enables information sharing with Federal government systems and across State and local jurisdictions.
In early July, the HSIN-Intelligence platform began establishing, in coordination with the Department of Justice and the Program Manager for the Information Sharing Environment, federated access capabilities across a number of other information sharing platforms such as Law Enforcement Online (LEO) and the Regional Information Sharing System Network (RISSNet). For the first time, DHS has allowed appropriate Department of Justice users of these other platforms direct access to DHS finished intelligence products residing on HSIN-Intelligence without requiring separate password or login requirements. By making access to multiple systems easier, we hope to reduce the gaps in knowledge that might occur from accessing only one system.
In addition, DHS and its Federal partners have made significant progress in their efforts to coordinate an effective, unified strategy for the handling of suspicious activity reporting (SARs). Using SARs has been identified as a capability to begin to see seemingly disparate activities that, when overlapped, show a pattern and possible threats. By designing a system that incorporates procedures and actions that begin at the State, local, and tribal levels, and are supported at the Federal level, DHS’ ability to review, analyze, and further disseminate important information that is collected by non-Federal partners is significantly enhanced.
Across the country, DHS has worked closely with local jurisdictions, including Los Angeles, Miami, Boston and Chicago, to understand their approach to SARs and how to best promote cross-integration. By identifying “best practices” at the State, local, and tribal levels, the Federal partners are able to build a collaborative SAR approach.
All of the Federal activities in SARs continue to carefully maintain the balance between the protection of its citizens and the protection of its citizen’s privacy and civil liberties. There are standing working groups and committees involving the General Counsel, Privacy, and Civil Rights/Civil Liberties Offices, of the Federal Departments involved in suspicious activity reporting.
We must remember, however, that increased information sharing comes with responsibilities. We are ever mindful that all of these efforts discussed today must be conducted with civil liberties and privacy rights at the table. To that end, our Privacy and Civil Rights Offices have delivered training to all of our deployed officers and are working with Bureau of Justice Assistance and PM-ISE to develop training for State, local, and tribal representatives in the fusion centers. We are also developing Privacy Impact Assessments for these efforts.
While these particular accomplishments of these inter-agency organizations such as the NFCCG and the ITACG and these other examples of multi-agency collaboration are important in their own right, they are particularly notable because of the close relationships among DHS, FBI, DOJ, the DNI, the PM-ISE, and the many State and local leaders that make them possible. It is these relationships in conjunction with the framework that is being created that makes information sharing a routine occurrence rather than a special event.
Read Under Secretary Charles E. Allen’s Full Testimony
NIEM Case Study Ideas Needed!
The IJIS Institute is working on an ongoing project to compile and edit NIEM adoption and use case studies. These case studies are a great vehicle to help others understand and use NIEM. As we advocate the use of NIEM for information sharing, it is important that we also share how NIEM is being used and highlight NIEM success stories and lessons learned. The IJIS Institute needs your help in collecting and developing these case studies for a variety of uses and among various levels of government and private industry.
NIEM adoption and use case studies are published in the electronic newsletter NIEM News. The newsletter is posted to the www.niem.gov site and is sent via e-mail to a targeted mailing list of interested parties. Case studies may also be used to highlight successful NIEM adoption and use in flyers or other promotional materials to help those interested.
Do you have any real-world examples of your work highlighting NIEM in action that might work as case studies? If so, we would like to hear from you! Case studies are simple write-ups that focus on challenges, solutions, and results. An organization can simply provide the IJIS Institute with some bullets or background material that we can turn into narrative form, or you can write the narrative yourself using a simple template that we can provide.
Please contact Andrea Walter at the IJIS Institute, andrea.walter@ijis.org, with your case study ideas. We appreciate any help that you can provide us in crafting useable case studies to get the word out about NIEM success stories!
NIEM Adoption and Use: New York State—Development of NIEM 2.0-Conformant IEPD for the New York State Intra-State Criminal History Report (Rap Sheet) Project
Synopsis
The purpose of this case study is to highlight the success of the development of NIEM 2.0-Conformant Information Exchange Package Documentation (IEPD) for the New York State Intra-State Criminal History Report (Rap Sheet) Project.
Agency Overview
The Division of Criminal Justice Services (DCJS) is a multifunction criminal justice support agency with a variety of responsibilities, including collection and analysis of statewide crime data; operation of the DNA databank and criminal fingerprint files; administration of federal and state criminal justice funds; support of criminal justice-related agencies across the state; and administration of the state’s Sex Offender Registry, which allows anyone to research the status of an offender.
New York State Integrated Justice Advisory Board (IJAB) serves as the governance body for integrating justice systems. IJAB is composed of representatives from DCJS, New York State (NYS) Department of Correctional Services, Division of Parole, Division of State Police, and Office of Homeland Security. The Sub-Committee on XML Best Practices and Standards conducted a comprehensive evaluation to identify the pros and cons of the use of NIEM. The recommendation was to adopt NIEM 2.0 and extend this version to incorporate all extensions that are required to meet the NYS business needs. This version becomes the NYS canonical model.
Challenge
The New York State Division of Criminal Justice Services (DCJS) proposed the use of grant funds provided by the National Governors Association (NGA) to support the development of a NIEM 2.0-Conformant Information Exchange Package Documentation (IEPD) to be used in the exchange of the New York State Intra-State Criminal History Report (rap sheet). This IEPD development is a major undertaking, given the volume and complexity of the data involved in this exchange, and the lack of previous staff experience in using the NIEM.
This proposed project was an integral part of the division’s current effort to replace its existing legacy rap sheet with an XML version. This NIEM 2.0-conformant rap sheet will be implemented using a newly developed service-oriented architecture platform that was designed for the implementation of the eJusticeNY Integrated Justice Portal scheduled for release in mid-2008.
Solution
The funds appropriated to DCJS through the NGA and BJA were committed to developing a NIEM 2.0-conformant Information Exchange Package Documentation (IEPD) rap sheet used in the exchange of the New York Intra-State Criminal History Report (rap sheet). This is an integral part of a newly developed service-oriented architecture platform that was designed for the implementation of eJusticeNY Integrated Justice Portal. The IEPD was developed using established methodology, including JIEM business analysis and model, UML modeling, mapping rap sheet data elements to NIEM, development of XML schemas, and publication of IEPD artifacts and schemas to the U.S. Department of Justice IEPD Clearinghouse.
Results
Results of this project include creation of a NIEM 2.0-conformant rap sheet IEPD, completion of all required IEPD artifacts, and publication to the U.S. Department of Justice IEPD Clearinghouse for use by other states. DCJS has also partnered with the New York City Mayor’s Office of the Criminal Justice Coordinator and other city-based criminal justice agencies to test and pilot an electronic XML rap sheet exchange between New York City via the DCJS portal. Other results include performance metrics to determine the length of time for developing a new IEPD and the cost of related activity, as well as identifying strategies for successfully bringing together different agencies and disciplines for collaborative team efforts.
Featured FAQ
What are NIEM Universal and Common Core components?
The NIEM data model includes two categories of reusable components (aka business objects)—Core components and domain-specific components. The NIEM Core components (Core Namespace) are further classified as either Universal or Common. Core components that are commonly shared and understood among all NIEM domains are classified as universal components (e.g., person, address, organization). Core components used in exchanges between multiple domains, but not universally shared, are identified as common components. Components understood and managed by a specific community of interest are considered domain-specific. Domain-specific components can extend Core components and must conform to the NIEM Naming and Design Rules.
NIEM Clearinghouse Update
Total IEPDs Submitted in July 2008
3 (NIEM), 0 (GJXDM)
Total IEPDs Posted
144 (68 GJXDM, 76 NIEM)
Summary of IEPDs for July
Wisconsin Office of Justice Assistance (IEPD related to drug case investigation)
- Drug Case Investigation Exchange IEPD
Colorado Integrated Criminal Justice Information System (CICJIS)
- Transport Order IEPD
Alabama Criminal Justice Information Center
- CONNECT Driver License Search IEPD v1.00
Status Updates and Changes on IEPDs
IJIS Institute (update on one IEPD—SAR for Local and State Entities)
- SAR for Local and State Entities IEPD v1.01
Program Manager, Information Sharing Environment (PM-ISE)
- ISE-SAR 1.0 IEPD
Upcoming NIEM Trainings
- September 16–18, 2008: Crime Net—Minnesota
- September 23–25, 2008: NIEM National Training—Ashburn, Virginia
- October 7–9, 2008: State Courts—Washington
- October 29–31, 2008: New York State Division of Criminal Justice Services (DCJS) in Troy, NY
- November 4–6, 2008: Department of Criminal Justice—Alabama