Practical XML: XPath

Originally published: 2009-01-01

Last updated: 2015-04-27

XPath is a way to navigate through an XML document, modeled on filesystem paths. An XPath expression consistes of a series of slash-separated “node tests” that describe how to traverse from one element to another within a document. At the most basic, the expression might be a series of element names. For example, given the following XML, the path “/foo/bar/baz” selects the single “baz” node:

<foo>
    <bar name='argle'>
        Argle
    </bar>
    <bar name='bargle'>
        Bargle
        <baz>Baz</baz>
    </bar>
</foo>

XPaths share more features with filesystem paths. They can be relative: if you have a reference to node foo, you can use the relative path “bar/baz” to get to baz. And there are special path components “.” and “..”, which refer to the current node and its parent. So, given a reference to either of the bar nodes, you can get to baz with “../bar/baz”.

Where XPath breaks from the filesystem model is that a “node test” is truly a test: it can contain a logical expression — a predicate — enclosed within square braces. Only those nodes that match the predicate will be selected. For example, the XML above has two nodes named “bar”. If you want to ensure that you select only the first, you could use a positional predicate: “/foo/bar[1]”. Or, you could use a predicate that tests the name attribute: “/foo/bar[@name='argle']”. You can also combine predicates: “/foo/bar[2][@name='argle']” is legal, although it won't select any nodes from the sample document.

The XPath specification will tell you all of the variants of an XPath expression. This article is about how to evaluate XPaths in Java. So here's the code to execute the first example above:

Document dom = // however you parse the document
XPath xpath = XPathFactory.newInstance().newXPath();
String result = xpath.evaluate("/foo/bar/baz", dom);

Pretty simple, eh? Like everything in the javax.xml hierarchy, it uses a factory: XPath is an interface, and the actual implementation class will depend on your JRE (OpenJDK uses the Apache Xerces implementation). Also, like the other packages in javax.xml, as you start to use more of the features of XPath the complexity (and code required) increases dramatically.

Result Types

The first piece of complexity is what, exactly, you get when evaluating an XPath expression. As you can see above, evaluate() returns a String. What if the path selected multiple nodes? Or selected a node that didn't have any content? Or failed to select any node? Here is a table showing the result of a few different expressions; the quotes are added to show the boundaries of the result.

XPath Result
/foo/bar/baz
Baz
/foo/bar
Argle   
/foo
    Argle      Bargle       Baz   
/fizzle

And now for what's happening behind the curtain. An XPath expression selects some set of nodes, which may be empty. The evaluate(String,Node) function returns the text content of the first selected node, or an empty string if no nodes were selected. “Text content” is equivalent to calling Node.getTextContent(): it returns the concatenation of all descendent text nodes. The path “/foo”, demonstrates this most dramatically: not only do we get the text from all nodes, but also all of the whitespace used to indent the document.

In many cases, the string value of an expression is all that you need, which is why the basic evaluate() method exists. If you need more control over your results, such as getting the actual nodes selected, there's a three-argument variant, evaluate(String,Node,QName), that lets you specify the result type using one of the values from XPathConstants. You can request a single node, multiple nodes, or translate the string value as a number or boolean. For example, to return all selected nodes:

NodeList nodes = (NodeList)xpath.evaluate("/foo/bar", dom, XPathConstants.NODESET);

Note that the constant name is NODESET but the actual return type is NodeList. This can be confusing: the result is most definitely not a java.util.Set; it can hold duplicate elements. Nor is it a java.util.List. It's an XML-specific type defined by the DOM Level 1 spec as holding an ordered list of nodes. The XPath spec, however, had a different group of people working on it, and they chose the term “node set” to refer to the same thing.

In practice, NodeList is clumsy: you have to access items by index, rather than using an iterator. As an alternative, the Practical XML library provides NodeListIterable, which wraps a NodeList for use with a Java 1.5 for-each loop.

Namespaces

Namespaces are perhaps the biggest source of pain in working with XPath. To see why, consider the following XML. It looks a lot like the example at the top of this article, with the addition of a default namespace. You might think that the example XPath expressions would work unchanged. But depending on how you configured your DocumentBuilder, you might be wrong.

<foo xmlns='http://www.example.com/example'>
    <bar name='argle'>
        Argle
    </bar>
    <bar name='bargle'>
        Bargle
        <baz>Baz</baz>
    </bar>
</foo>

By default, DocumentBuilderFactory creates XML parsers that are not namespace aware. And if you have a single namespace for an entire document, as in this example, you can safely use the default parser configuration and write simple XPaths. In fact, that's what I recommend in most cases where people run into namespace problems with XPath.

But sometimes, you have to deal with documents that use multiple namespaces, where those namespaces are used to identify different types of data. Consider an order from an eCommerce site: the order might include a state element that describes the order status (received, processing, shipped, &c), and this is very different than state in the customer's address. A good design will use separate namespaces for these values, but if you parse without namespaces and use an XPath like “//state”, you'll be in trouble. So, to avoid bugs, you enable namespace-aware parsing, but then a simple path expression no longer selects anything.

InputStream xml = // ...

DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
DocumentBuilder db = dbf.newDocumentBuilder();
Document dom = db.parse(new InputSource(xml));

XPath xpath = XPathFactory.newInstance().newXPath();
String result = xpath.evaluate("/foo/bar/baz", dom);

The reason: XPath is fully namespace aware, and node tests match both localname and namespace. If you don't specify a namespace in the path, the evaluator assumes that the node doesn't have a namespace. Making life difficult, there's no way to explicitly specify a namespace as part of the node test; you must instead use a “qualified name” name (prefix:localname), and provide an external mapping from prefix to namespace.

So, to select node baz, we first have to add a prefix to each node in the expression: “/ns:foo/ns:bar/ns:baz”. And then, we have create a NamespaceContext object to tell the evaluator the namespace bound to that prefix:

XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new NamespaceContext()
{
    private String _prefix = "ns";
    private String _namespaceUri = "http://www.example.com/example";
    private List<String> _prefixes = Arrays.asList(_prefix);


    @Override
    @SuppressWarnings("rawtypes")
    public Iterator getPrefixes(String uri)
    {
        if (uri == null)
            throw new IllegalArgumentException("nsURI may not be null");
        else if (_namespaceUri.equals(uri))
            return _prefixes.iterator();
        else if (XMLConstants.XML_NS_URI.equals(uri))
            return Arrays.asList(XMLConstants.XML_NS_PREFIX).iterator();
        else if (XMLConstants.XMLNS_ATTRIBUTE_NS_URI.equals(uri))
            return Arrays.asList(XMLConstants.XMLNS_ATTRIBUTE).iterator();
        else
            return Collections.emptyList().iterator();
    }


    @Override
    public String getPrefix(String uri)
    {
        if (uri == null)
            throw new IllegalArgumentException("nsURI may not be null");
        else if (_namespaceUri.equals(uri))
            return _prefix;
        else if (XMLConstants.XML_NS_URI.equals(uri))
            return XMLConstants.XML_NS_PREFIX;
        else if (XMLConstants.XMLNS_ATTRIBUTE_NS_URI.equals(uri))
            return XMLConstants.XMLNS_ATTRIBUTE;
        else
            return null;
    }


    @Override
    public String getNamespaceURI(String prefix)
    {
        if (prefix == null)
            throw new IllegalArgumentException("prefix may not be null");
        else if (_prefix.equals(prefix))
            return _namespaceUri;
        else if (XMLConstants.XML_NS_PREFIX.equals(prefix))
            return XMLConstants.XML_NS_URI;
        else if (XMLConstants.XMLNS_ATTRIBUTE.equals(prefix))
            return XMLConstants.XMLNS_ATTRIBUTE_NS_URI;
        else
            return null;
    }
});

String result = xpath.evaluate("/ns:foo/ns:bar/ns:baz", dom);

That was probably much more code than you expected. A lot of it is boilerplate for reserved prefixes, which in my experience is never called when evaluating a simple XPath. However, rather than simply omitting that code, I recommend implementing it in a base class that's extended for your specific namespaces. Or, turn to the Practical XML library, which provides NamespaceResolver for documents that use multiple namespaces, and SimpleNamespaceResolver for documents that only use a single namespace.

XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new SimpleNamespaceResolver("ns", "http://www.example.com/example"));

String result = xpath.evaluate("/ns:foo/ns:bar/ns:baz", dom);

Variables

So far, all of the examples have used literal strings containing the XPath expression. Doing this opens the door for hackers: an XPath expression built directly from client data is subject to “XPath injection,” similar to the better-known attacks against SQL. And, just as most SQL injection attacks can be thwarted through use of parameterized prepared statements rather than literal SQL, XPath expressions can be protected through the use of variables.

XPath variables appear in an expression as a dollar-sign followed by an identifier. For example, “/foo/bar[@name=$name]”. To provide evaluation-time values for variables, you must attach a variable resolver to your XPath object:

XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setXPathVariableResolver(new XPathVariableResolver()
{
    @Override
    public Object resolveVariable(QName var)
    {
        if (var.getLocalPart().equals("name"))
            return "argle";
        else
            return "";
    }
});

String result = xpath.evaluate("//bar[@name=$name]", dom);

First thing to note is that the variable name is a qualified name: it can have a namespace. I don't think that namespaced variables are a very good idea: aside from possibly buggy behavior in the JDK's evaluator, I see them as an indication that your code is trying to do too much.

But, namespaced or not, how should you implement your resolver? I show a simple if-else chain, which is fine if you have only a few variables. If you have more, I recommend using a Map<QName,String>; this is one case where the Practical XML library does not have a drop-in replacement.

Functions

Functions in an XPath expression look a lot like functions in other programming languages: a name, followed by a list of arguments in parentheses. An argument could be literal text, another function call, or (in many cases) an XPath expression. For example, the following expression uses the built-in translate() function to select a node and return its content as uppercase — provided that it contains only US-ASCII characters:

translate(string(/foo/bar/baz), 'abcdefghijklmnopqrstuvwxyz', 'ABCDEFGHIJKLMNOPQRSTUVWXYZ')

This expression is, quite frankly, ugly: it buries the node selection in a mass of text. It's also flawed, in that it translates a limited set of characters into another limited set of characters. If we need to uppercase text, it would be better to use the Java method String.toUpperCase(), which will do a locale-correct translation. And with add-on functions we can do just that … although after you see the work involved, you might have second thoughts about doing so.

The first part of implementing a user-defined function is easy: implement the function. Actually, it's not so easy: XPath functions are instances of the XPathFunction interface, which provides a single evaluate() method that receives a list of objects. You have to figure out the type of each item in that list and handle it appropriately.

To uppercase a string, you first need a string. However, an XPath function argument may be any valid XPath expression: a string, number, boolean, or nodeset. We'll handle strings and nodesets, and follow the convention of taking the text value of the first element of the nodeset. We'll throw if you construct an XPath expression that doesn't return one of these two; it makes little sense to uppercase numbers or booleans (although in production code I would return the value).

private static class Uppercase implements XPathFunction
{
    @Override
    @SuppressWarnings("rawtypes")
    public Object evaluate(List args) throws XPathFunctionException
    {
        Object arg = args.iterator().next();
        if (arg instanceof NodeList)
        {
            NodeList nodes = (NodeList)arg;
            if (nodes.getLength() == 0)
                return "";
            else
                return nodes.item(0).getTextContent().toUpperCase();
        }
        else if (arg instanceof String)
        {
            return ((String)arg).toUpperCase();
        }
        else
            throw new XPathFunctionException("invalid argument: " + arg);
    }
}

Once you've implemented your function(s), you use an XPathFunctionResolver to tell the XPath evaluator to use them:

XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new SimpleNamespaceResolver("fn", "urn:x-com.kdgregory.example.xpath.functions"));
xpath.setXPathFunctionResolver(new XPathFunctionResolver()
{
    private QName myFunctionName = new QName("urn:x-com.kdgregory.example.xpath.functions", "uppercase");
    private XPathFunction myFunction = new Uppercase();

    @Override
    public XPathFunction resolveFunction(QName functionName, int arity)
    {
        if (functionName.equals(myFunctionName) && (arity == 1))
            return myFunction;
        else
            return null;
    }
});

There's a lot of stuff happening here. First off, functions are identified by qualified name, and unlike variables, you must use a namespace. So, first step is to register a namespace context that includes our namespace, identified by the prefix “fn” (and I've used the utility function from PracticalXML to do so).

When the XPath evaluator sees a function that it doesn't recognize (ie, not defined by the spec), it turns to the function resolver. As you can see, the resolver takes two parameters: the first is the function name, and the second is how many arguments the evaluator will pass to that function. While you could get away with just checking function name in most situations, the additional arity check ensures that you don't have any unexpected typos (like a missing close parenthesis) in your paths.

As with my variable resolver example, I'm using a simple if-else chain to check the function. While a Map would be simpler if we were just matching on name, the additional arity check makes it less suitable. If there's no match for the function name, returning null tells the evaluator to throw an exception.

With all that done, now you can use your function. Here are two examples, one which uppercases the returned text, and one that uses the uppercase as part of a node test.

String result1 = xpath.evaluate("fn:uppercase(//baz)", dom);

String result2 = xpath.evaluate("//bar[fn:uppercase(@name)='BARGLE']", dom);

As you might guess, the Practical XML library has a few classes to make functions easier. FunctionResolver implements an efficient lookup with arity, AbstractFunction is an abstract class for building functions, and there are several example function implementations.

To be honest, I believe that XPath functions are often more trouble than they're worth. Often it's simpler to just select a bunch of nodes, then apply tests or transform results using Java code.

XPathExpression

All of the examples so far have called the XPath.evaluate() method. As you might guess, this has to compile the expression into an internal form before using it. If you plan to reuse the expression, it makes sense to compile it once and save that time on future evaluations:

XPathExpression expr = xpath.compile("/foo/bar/baz");
// ...
String value = expr.evaluate(dom);

The XPathExpression object provides all of the evaluate() methods from XPath, but nothing else. You must fully configure the XPath object before you compile an expression from it.

XPathWrapper

Although the XPath interfaces provided with the JDK are individually simple, I find them painful in practice. Particularly when using namespaces. The fact that most of this pain comes from boilerplate code led to the development of XPathWrapper. This class internalizes all of the ancillary objects used by XPath, compiles its expression for reuse, and is constructed using the “builder” pattern:

XPathWrapper xpath = new XPathWrapper("//ns:bar[fn:uppercase(@name) = $var]")
                     .bindNamespace("ns", "http://www.example.com/example")
                     .bindNamespace("fn", "urn:x-com.kdgregory.example.xpath.functions")
                     .bindFunction(new QName("urn:x-com.kdgregory.example.xpath.functions", "uppercase"), new Uppercase(), 1)
                     .bindVariable("var", "BARGLE");

String result = xpath.evaluateAsString(dom);

If you need to create multiple XPath expressions using similar configuration, XPathWrapperFactory will let you do that. It will also cache the generated wrappers for reuse.

JxPath

While this article has focused on the JDK's implementation of XPath, it is not the only implementation available. Of the alternatives, Apache Commons JxPath is notable because it allows XPath expressions to be used with arbitrary object graphs, not just XML. Out of the box it supports bean-style objects, JDK collections, and the context objects provided to J2EE servlets.

While all that is useful, what's nicer is that JxPath is easy to extend. For example, you implement functions using a single class, none of which need to be attached to a namespace. Once you register the class, JxPath will use reflection to find the function.

You can also easily extend JxPath to access arbitrary structured data types. I used it for a project where we stored content in the database as a collection of IDs: each unique string had its own ID, significantly reducing the storage of duplicate data. Since the actual data tables were just collections of IDs, we used JxPath to implement human-readable queries, by adding an override to JxPath's node test and navigation code.

XPath as Debugging Tool

One of the problems of working with a DOM document is tracing issues back to their source. It's one reason that I recommend validating at the time of parsing, because once the parsing is done you've lost any source line or column numbers.

However, you can create an XPath expression that will uniquely identify any node in an XML document, using predicates to differentiate nodes with the same name. There are some caveats when doing so: in particular, keeping track of namespaces and providing any new bindings to the calling code.

Not surprisingly, the Practical XML library provides methods to do just this: DomUtil.getAbsolutePath() returns a path with positional predicates, that can be used to select the same node in the future. There's also DomUtil.getPath(), which returns a path with attribute predicates. This variant is unlikely to be useful for selecting the same node later, but I find it very useful as a debugging tool.

For More Information

The XPath 1.0 spec is actually pretty readable as W3C specs go.

The Practical XML library provides utilities for working with XML in many different ways, not just XPath evaluation. Be aware that the latest compiled version is not available for download from SourceForge; instead, get it from the Maven central repository.

You can download all examples for this article as a Maven project in a JARfile. If you want to look at specific examples, click the following links:

Copyright © Keith D Gregory, all rights reserved

This site does not intentionally use tracking cookies. Any cookies have been added by my hosting provider (InMotion Hosting), and I have no ability to remove them. I do, however, have access to the site's access logs with source IP addresses.