Practical XML: XPath
Originally published: 2009-01-01
Last updated: 2015-04-27
XPath is a way to navigate through an XML document, modeled on filesystem paths.
An XPath expression consistes of a series of slash-separated “node tests”
that describe how to traverse from one element to another within a document. At
the most basic, the expression might be a series of element names. For example,
given the following XML, the path “/foo/bar/baz” selects the
single “baz” node:
<foo>
<bar name='argle'>
Argle
</bar>
<bar name='bargle'>
Bargle
<baz>Baz</baz>
</bar>
</foo>
XPaths share more features with filesystem paths. They can be relative: if
you have a reference to node foo, you can use the relative
path “bar/baz” to get to baz. And there
are special path components “.” and “..”,
which refer to the current node and its parent. So, given a reference to
either of the bar nodes, you can get to
baz with “../bar/baz”.
Where XPath breaks from the filesystem model is that a “node test”
is truly a test: it can contain a logical expression — a predicate —
enclosed within square braces. Only those nodes that match the predicate will
be selected. For example, the XML above has two nodes named “bar”.
If you want to ensure that you select only the first, you could use a positional
predicate: “/foo/bar[1]”. Or, you could use a predicate that
tests the name attribute: “/foo/bar[@name='argle']”.
You can also combine predicates: “/foo/bar[2][@name='argle']” is legal, although it won't select any nodes from the sample document.
The XPath specification will tell you all of the variants of an XPath expression. This article is about how to evaluate XPaths in Java. So here's the code to execute the first example above:
Document dom = // however you parse the document
XPath xpath = XPathFactory.newInstance().newXPath();
String result = xpath.evaluate("/foo/bar/baz", dom);
Pretty simple, eh? Like everything in the javax.xml hierarchy,
it uses a factory: XPath is an
interface, and the actual implementation class will depend on your JRE (OpenJDK
uses the Apache Xerces implementation). Also, like the other packages in
javax.xml, as you start to use more of the features of XPath
the complexity (and code required) increases dramatically.
Result Types
The first piece of complexity is what, exactly, you get when evaluating
an XPath expression. As you can see above, evaluate() returns
a String. What if the path selected multiple nodes? Or selected
a node that didn't have any content? Or failed to select any node? Here is a
table showing the result of a few different expressions; the quotes are added
to show the boundaries of the result.
| XPath | Result |
|---|---|
/foo/bar/baz |
“ |
/foo/bar |
“ |
/foo |
“ |
/fizzle |
“ |
And now for what's happening behind the curtain. An XPath expression selects some
set of nodes, which may be empty. The evaluate(String,Node) function returns the text content of the first selected node, or an empty
string if no nodes were selected. “Text content” is equivalent to
calling Node.getTextContent(): it
returns the concatenation of all descendent text nodes. The path
“/foo”, demonstrates this most dramatically: not only do
we get the text from all nodes, but also all of the whitespace used to indent
the document.
In many cases, the string value of an expression is all that you need, which is
why the basic evaluate() method exists. If you need more
control over your results, such as getting the actual nodes selected, there's a
three-argument variant, evaluate(String,Node,QName), that
lets you specify the result type using one of the values from
XPathConstants.
You can request a single node, multiple nodes, or translate the string value as a
number or boolean. For example, to return all selected nodes:
NodeList nodes = (NodeList)xpath.evaluate("/foo/bar", dom, XPathConstants.NODESET);
Note that the constant name is NODESET but the actual return
type is NodeList. This can be confusing:
the result is most definitely not a java.util.Set;
it can hold duplicate elements. Nor is it a java.util.List.
It's an XML-specific type defined by the
DOM Level 1 spec as holding an ordered list of nodes. The XPath spec,
however, had a different group of people working on it, and they chose the
term “node set” to refer to the same thing.
In practice, NodeList is clumsy: you have to access items
by index, rather than using an iterator. As an alternative, the Practical XML
library provides NodeListIterable,
which wraps a NodeList for use with a Java 1.5 for-each loop.
Namespaces
Namespaces are perhaps the biggest source of pain in working with XPath. To
see why, consider the following XML. It looks a lot like the example at the
top of this article, with the addition of a default namespace. You might
think that the example XPath expressions would work unchanged. But depending
on how you configured your DocumentBuilder, you might be
wrong.
<foo xmlns='http://www.example.com/example'>
<bar name='argle'>
Argle
</bar>
<bar name='bargle'>
Bargle
<baz>Baz</baz>
</bar>
</foo>
By default, DocumentBuilderFactory creates XML parsers that
are not namespace aware. And if you have a single namespace for an entire
document, as in this example, you can safely use the default parser configuration
and write simple XPaths. In fact, that's what I recommend in most cases where
people run into namespace problems with XPath.
But sometimes, you have to deal with documents that use multiple namespaces,
where those namespaces are used to identify different types of data. Consider an
order from an eCommerce site: the order might include a state
element that describes the order status (received, processing, shipped, &c),
and this is very different than state in the customer's address.
A good design will use separate namespaces for these values, but if you parse
without namespaces and use an XPath like “//state”, you'll
be in trouble. So, to avoid bugs, you enable namespace-aware parsing, but then
a simple path expression no longer selects anything.
InputStream xml = // ...
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
DocumentBuilder db = dbf.newDocumentBuilder();
Document dom = db.parse(new InputSource(xml));
XPath xpath = XPathFactory.newInstance().newXPath();
String result = xpath.evaluate("/foo/bar/baz", dom);
The reason: XPath is fully namespace aware, and node tests match both localname
and namespace. If you don't specify a namespace in the path, the evaluator assumes
that the node doesn't have a namespace. Making life difficult, there's no way to
explicitly specify a namespace as part of the node test; you must instead use a
“qualified name” name (prefix:localname), and
provide an external mapping from prefix to namespace.
So, to select node baz, we first have to add a prefix to each
node in the expression: “/ns:foo/ns:bar/ns:baz”. And then, we have
create a NamespaceContext object to tell the evaluator the namespace bound to that prefix:
XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new NamespaceContext()
{
private String _prefix = "ns";
private String _namespaceUri = "http://www.example.com/example";
private List<String> _prefixes = Arrays.asList(_prefix);
@Override
@SuppressWarnings("rawtypes")
public Iterator getPrefixes(String uri)
{
if (uri == null)
throw new IllegalArgumentException("nsURI may not be null");
else if (_namespaceUri.equals(uri))
return _prefixes.iterator();
else if (XMLConstants.XML_NS_URI.equals(uri))
return Arrays.asList(XMLConstants.XML_NS_PREFIX).iterator();
else if (XMLConstants.XMLNS_ATTRIBUTE_NS_URI.equals(uri))
return Arrays.asList(XMLConstants.XMLNS_ATTRIBUTE).iterator();
else
return Collections.emptyList().iterator();
}
@Override
public String getPrefix(String uri)
{
if (uri == null)
throw new IllegalArgumentException("nsURI may not be null");
else if (_namespaceUri.equals(uri))
return _prefix;
else if (XMLConstants.XML_NS_URI.equals(uri))
return XMLConstants.XML_NS_PREFIX;
else if (XMLConstants.XMLNS_ATTRIBUTE_NS_URI.equals(uri))
return XMLConstants.XMLNS_ATTRIBUTE;
else
return null;
}
@Override
public String getNamespaceURI(String prefix)
{
if (prefix == null)
throw new IllegalArgumentException("prefix may not be null");
else if (_prefix.equals(prefix))
return _namespaceUri;
else if (XMLConstants.XML_NS_PREFIX.equals(prefix))
return XMLConstants.XML_NS_URI;
else if (XMLConstants.XMLNS_ATTRIBUTE.equals(prefix))
return XMLConstants.XMLNS_ATTRIBUTE_NS_URI;
else
return null;
}
});
String result = xpath.evaluate("/ns:foo/ns:bar/ns:baz", dom);
That was probably much more code than you expected. A lot of it is boilerplate
for reserved prefixes, which in my experience is never called when evaluating
a simple XPath. However, rather than simply omitting that code, I recommend
implementing it in a base class that's extended for your specific namespaces.
Or, turn to the Practical XML library, which provides
NamespaceResolver for documents that use multiple namespaces, and
SimpleNamespaceResolver for documents that only use a single namespace.
XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new SimpleNamespaceResolver("ns", "http://www.example.com/example"));
String result = xpath.evaluate("/ns:foo/ns:bar/ns:baz", dom);
Variables
So far, all of the examples have used literal strings containing the XPath expression. Doing this opens the door for hackers: an XPath expression built directly from client data is subject to “XPath injection,” similar to the better-known attacks against SQL. And, just as most SQL injection attacks can be thwarted through use of parameterized prepared statements rather than literal SQL, XPath expressions can be protected through the use of variables.
XPath variables appear in an expression as a dollar-sign followed by an
identifier. For example, “/foo/bar[@name=$name]”. To
provide evaluation-time values for variables, you must attach a variable
resolver to your XPath object:
XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setXPathVariableResolver(new XPathVariableResolver()
{
@Override
public Object resolveVariable(QName var)
{
if (var.getLocalPart().equals("name"))
return "argle";
else
return "";
}
});
String result = xpath.evaluate("//bar[@name=$name]", dom);
First thing to note is that the variable name is a qualified name: it can have a namespace. I don't think that namespaced variables are a very good idea: aside from possibly buggy behavior in the JDK's evaluator, I see them as an indication that your code is trying to do too much.
But, namespaced or not, how should you implement your resolver? I show a simple
if-else chain, which is fine if you have only a few variables.
If you have more, I recommend using a Map<QName,String>; this
is one case where the Practical XML library does not have a drop-in replacement.
Functions
Functions in an XPath expression look a lot like functions in other
programming languages: a name, followed by a list of arguments in
parentheses. An argument could be literal text, another function call, or
(in many cases) an XPath expression. For example, the following expression
uses the built-in translate() function to select a
node and return its content as uppercase — provided that it contains
only US-ASCII characters:
translate(string(/foo/bar/baz), 'abcdefghijklmnopqrstuvwxyz', 'ABCDEFGHIJKLMNOPQRSTUVWXYZ')
This expression is, quite frankly, ugly: it buries the node selection in
a mass of text. It's also flawed, in that it translates a limited set of
characters into another limited set of characters. If we need to uppercase
text, it would be better to use the Java method String.toUpperCase(),
which will do a locale-correct translation. And with add-on functions we can
do just that … although after you see the work involved, you might
have second thoughts about doing so.
The first part of implementing a user-defined function is easy: implement
the function. Actually, it's not so easy: XPath functions are instances of
the XPathFunction interface, which
provides a single evaluate() method that receives a list
of objects. You have to figure out the type of each item in that list and
handle it appropriately.
To uppercase a string, you first need a string. However, an XPath function argument may be any valid XPath expression: a string, number, boolean, or nodeset. We'll handle strings and nodesets, and follow the convention of taking the text value of the first element of the nodeset. We'll throw if you construct an XPath expression that doesn't return one of these two; it makes little sense to uppercase numbers or booleans (although in production code I would return the value).
private static class Uppercase implements XPathFunction
{
@Override
@SuppressWarnings("rawtypes")
public Object evaluate(List args) throws XPathFunctionException
{
Object arg = args.iterator().next();
if (arg instanceof NodeList)
{
NodeList nodes = (NodeList)arg;
if (nodes.getLength() == 0)
return "";
else
return nodes.item(0).getTextContent().toUpperCase();
}
else if (arg instanceof String)
{
return ((String)arg).toUpperCase();
}
else
throw new XPathFunctionException("invalid argument: " + arg);
}
}
Once you've implemented your function(s), you use an
XPathFunctionResolver to tell
the XPath evaluator to use them:
XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new SimpleNamespaceResolver("fn", "urn:x-com.kdgregory.example.xpath.functions"));
xpath.setXPathFunctionResolver(new XPathFunctionResolver()
{
private QName myFunctionName = new QName("urn:x-com.kdgregory.example.xpath.functions", "uppercase");
private XPathFunction myFunction = new Uppercase();
@Override
public XPathFunction resolveFunction(QName functionName, int arity)
{
if (functionName.equals(myFunctionName) && (arity == 1))
return myFunction;
else
return null;
}
});
There's a lot of stuff happening here. First off, functions are identified
by qualified name, and unlike variables, you must use a namespace.
So, first step is to register a namespace context that includes our
namespace, identified by the prefix “fn” (and I've used
the utility function from PracticalXML to do so).
When the XPath evaluator sees a function that it doesn't recognize (ie, not defined by the spec), it turns to the function resolver. As you can see, the resolver takes two parameters: the first is the function name, and the second is how many arguments the evaluator will pass to that function. While you could get away with just checking function name in most situations, the additional arity check ensures that you don't have any unexpected typos (like a missing close parenthesis) in your paths.
As with my variable resolver example, I'm using a simple if-else chain to
check the function. While a Map would be simpler if we
were just matching on name, the additional arity check makes it less suitable.
If there's no match for the function name, returning null tells the evaluator to throw an exception.
With all that done, now you can use your function. Here are two examples, one which uppercases the returned text, and one that uses the uppercase as part of a node test.
String result1 = xpath.evaluate("fn:uppercase(//baz)", dom);
String result2 = xpath.evaluate("//bar[fn:uppercase(@name)='BARGLE']", dom);
As you might guess, the Practical XML library has a few classes to make functions easier.
FunctionResolver implements an efficient lookup with arity,
AbstractFunction is an abstract class for building functions,
and there are several example function implementations.
To be honest, I believe that XPath functions are often more trouble than they're worth. Often it's simpler to just select a bunch of nodes, then apply tests or transform results using Java code.
XPathExpression
All of the examples so far have called the XPath.evaluate() method. As you might guess, this has to compile the expression into an
internal form before using it. If you plan to reuse the expression, it
makes sense to compile it once and save that time on future evaluations:
XPathExpression expr = xpath.compile("/foo/bar/baz");
// ...
String value = expr.evaluate(dom);
The XPathExpression object provides
all of the evaluate() methods from XPath,
but nothing else. You must fully configure the XPath object
before you compile an expression from it.
XPathWrapper
Although the XPath interfaces provided with the JDK are individually
simple, I find them painful in practice. Particularly when using
namespaces. The fact that most of this pain comes from boilerplate
code led to the development of
XPathWrapper.
This class internalizes all of the ancillary objects used by
XPath, compiles its expression for reuse, and is constructed
using the “builder” pattern:
XPathWrapper xpath = new XPathWrapper("//ns:bar[fn:uppercase(@name) = $var]")
.bindNamespace("ns", "http://www.example.com/example")
.bindNamespace("fn", "urn:x-com.kdgregory.example.xpath.functions")
.bindFunction(new QName("urn:x-com.kdgregory.example.xpath.functions", "uppercase"), new Uppercase(), 1)
.bindVariable("var", "BARGLE");
String result = xpath.evaluateAsString(dom);
If you need to create multiple XPath expressions using similar configuration,
XPathWrapperFactory will let you do that. It will also cache the generated wrappers for reuse.
JxPath
While this article has focused on the JDK's implementation of XPath, it is not the only implementation available. Of the alternatives, Apache Commons JxPath is notable because it allows XPath expressions to be used with arbitrary object graphs, not just XML. Out of the box it supports bean-style objects, JDK collections, and the context objects provided to J2EE servlets.
While all that is useful, what's nicer is that JxPath is easy to extend. For example, you implement functions using a single class, none of which need to be attached to a namespace. Once you register the class, JxPath will use reflection to find the function.
You can also easily extend JxPath to access arbitrary structured data types. I used it for a project where we stored content in the database as a collection of IDs: each unique string had its own ID, significantly reducing the storage of duplicate data. Since the actual data tables were just collections of IDs, we used JxPath to implement human-readable queries, by adding an override to JxPath's node test and navigation code.
XPath as Debugging Tool
One of the problems of working with a DOM document is tracing issues back to their source. It's one reason that I recommend validating at the time of parsing, because once the parsing is done you've lost any source line or column numbers.
However, you can create an XPath expression that will uniquely identify any node in an XML document, using predicates to differentiate nodes with the same name. There are some caveats when doing so: in particular, keeping track of namespaces and providing any new bindings to the calling code.
Not surprisingly, the Practical XML library provides methods to do just this:
DomUtil.getAbsolutePath()
returns a path with positional predicates, that can be used to select the
same node in the future. There's also DomUtil.getPath(),
which returns a path with attribute predicates. This variant is unlikely to be
useful for selecting the same node later, but I find it very useful as a
debugging tool.
For More Information
The XPath 1.0 spec is actually pretty readable as W3C specs go.
The Practical XML library provides utilities for working with XML in many different ways, not just XPath evaluation. Be aware that the latest compiled version is not available for download from SourceForge; instead, get it from the Maven central repository.
You can download all examples for this article as a Maven project in a JARfile. If you want to look at specific examples, click the following links:
- SimpleXPathExample parses an in-program XML document and applies an XPath to the DOM.
- CompiledXPathExample is the same program, with the additional step of compiling the XPath expression.
- NamespacedXPathExample shows how to apply XPath to XML documents with namespace, using the tools provided by the JDK.
- SimpleNamespacedXPathExample does the same thing, but uses the
SimpleNamespaceResolverfrom PracticalXML to eliminate several dozen lines of code. To compile and run, you must have the PracticalXML libary on your classpath. - FunctionExample shows the JDK approach to defining an
uppercasefunction and using it in XPath expressions. - VariableExample creates an
XPathVariableResolverand shows the use of variables in an expression. - PracticalXmlExample demonstrates the use of PracticalXML's
XPathWrapperclass to evaluate XPath expressions with namespaces, functions, and variables. Again, you must have the library in your classpath to compile and run this example.
Copyright © Keith D Gregory, all rights reserved