Introduction

The SmartComponent Library integrates with Elasticsearch to provide full text, keyword and phonetic search across OpenEdge database data. Data is indexed in Elasticsearch and can then be searched with a speed and a fuzziness that a relational query cannot deliver.

The integration has two sides: the uploader writes the data of a Business Entity into an index, the query engine queries an index and reads the matching documents back into an ABL temp-table.

The uploader is built on the Consultingwerk.SmartIndex .NET assembly, which ships with the setup and is registered in Assemblies/assemblies.xml. It provides the low level Elasticsearch client, the conversion of a JSON document dump into the Elasticsearch bulk format, and the phonetic analysis (Metaphone, Soundex, Cologne / Kölner Phonetik). The query engine is pure ABL and runs on PASOE and Linux as well.

Sub pages

Elasticsearch Data Uploader (upload-data.cmd) — performing the initial, full upload of a Business Entity's data into an index.
File based configuration for upload-data.cmd — providing the uploader's arguments in an .elasticupload JSON file instead of on the command line.
Elasticsearch Query Engine — querying an index from ABL, the operator mapping, paging and aggregations.
Elasticsearch Index Statistics (index-stats.cmd) — document counts, store size and search statistics of an index.

Components

Component	Description

Component	Description
`Consultingwerk.SmartIndex` (.NET)	The .NET assembly providing `SmartIndexer`, the bulk file conversion and the phonetic encoders.
`Consultingwerk.SmartIndex.ElasticSearch.ElasticSearchUploader`	The ABL class wrapping the .NET indexer. Dumps a ProDataset to JSON and uploads it into an Elasticsearch index.
`upload-data.cmd`	Command line utility performing the initial (full) upload of a Business Entity's data into an Elasticsearch index.
`index-stats.cmd`	Command line utility reporting the document counts, the store size and the search statistics of an index.
`ISmartIndexQueryEngine` / `ISmartIndexQueryBuilder`	The query side: builds a query — as a Lucene query string or from a TableQuery — executes it and reads the hits into a temp-table, with paging and aggregations.

Configuration

The connection to the Elasticsearch cluster is configured in the .applicationsettings file (and in .restapplicationsettings for PASOE sessions) in the elasticsearch sub structure:

"elasticsearch": {
    "endpoint": "https://sfrbo.consultingwerkcloud.com:9200/",
    "user": "consultingwerk",
    "password": "elasticwerk"
},

Setting	Description

Setting	Description
`elasticsearch.endpoint`	The base URL of the Elasticsearch cluster, with or without a trailing slash. The development cluster is `https://sfrbo.consultingwerkcloud.com:9200/`.
`elasticsearch.user`	The user name used to authenticate against the cluster.
`elasticsearch.password`	The password used to authenticate against the cluster.

The settings are read through the IConfigurationProvider service and resolved through the PROPATH, so both the uploader and the query engine also work when they are driven from a project that has no settings file of its own — which is the case for the UnitTests project.

Field mapping

How a column is uploaded decides how it can be searched and whether it can be aggregated or sorted on. This matters more than it looks — aggregating a field that is not mapped as a keyword returns an empty result rather than an error.

Uploaded as	Mapping	Search on	Aggregate / sort on

Uploaded as	Mapping	Search on	Aggregate / sort on
`-keywordColumns`	`keyword`	the field name	the field name
`-phoneticColumns`	`text` with the phonetic analyzer, plus a `keyword` sub-field	the field name	`<field>.keyword`
neither	dynamic mapping — `text` plus a `keyword` sub-field	the field name	`<field>.keyword`

The keyword sub-field of a phonetic column was added later. An index uploaded before that has phonetic columns mapped as plain text, so aggregating them yields no buckets — such an index has to be uploaded again.

Cluster prerequisites

The phonetic analysis is provided by an Elasticsearch plugin that is not part of the default distribution. Without it, creating an index with phonetic columns fails with Unknown filter type [phonetic]:

bin/elasticsearch-plugin install analysis-phonetic

The plugin version has to match the Elasticsearch version exactly, and the node has to be restarted afterwards. Verify with bin/elasticsearch-plugin list or GET /_cat/plugins?v.

The cluster must accept TLS 1.2. The .NET Framework hosted by the AVM defaults ServicePointManager:SecurityProtocol to SSL 3.0 / TLS 1.0, so the uploader raises it to TLS 1.2 before the upload — a cluster offering only TLS 1.0/1.1 is not supported.

Automated tests

Consultingwerk.SmartIndexTests.SCL5577.SalesrepUploadTest in the UnitTests project performs the Sports2000 Salesrep upload against the configured cluster. It bypasses upload-data.p — the test runner has already connected the databases — fetches through the ServiceInterface and drives the ElasticSearchUploader directly. It uploads into a dedicated index, compares the number of fetched records against the document count reported by Elasticsearch, and removes the index again afterwards.

The suite is registered in unittests.xml as SmartIndex Tests (Consultingwerk.SmartIndexTests.*), which is what the Jenkins pipeline executes. The suite therefore needs the cluster to be reachable from the build node and the analysis-phonetic plugin to be installed on it.