Combining the results from 2 indexes in splunk query

Question

I have one index idx1 and other index idx2 and a common column "A" on which matching needs to be done.

I'm facing difficulty in combining the data from both the columns. I've to combine the data in such a way that if there is duplicate then the data from idx1 must be prioritized over data from idx2; i.e. basically equivalent of set operation [a+(b-a)].

I've tried the following :

| set diff  [ search index=idx2 sourcetype=src | dedup A ] [search index=idx1 sourcetype=src | dedup A ]
| stats count BY index A
| table index A

Here I get total 10840 statistics with both columns filled.

But when I want to display other columns from both the indexes I get empty columns for those.

Upon executing :

| set diff  [ search index=1idx1 sourcetype=src | dedup A ] [search index=idx2 sourcetype=src | dedup A ]
    | stats count BY index

I get the output as

index count idx1 4791 idx2 6049

Can anyone help me how should I proceed??

I've tried even this but not sure

index=idx1 sourcetype=src
| append [
| set diff  [ search index=idx2 sourcetype=src | dedup A ] [search index=idx1 sourcetype=src | dedup A ]]
    | stats count BY index A
    | table index A

Answer 1

I tried something like this :

index=idx1 sourcetype=src
| append [search index=idx2 sourcetype=src ]
| dedup A| table A B C D

idx1 had 4791 events

idx2 had 6049 events

(idx1-idx2) has 2590 events

(idx2-idx1) has 3848 events

union of 2 indexes has 8639 events

intersection of 2 indexes has 2201 events

So after executing the above query I got 8639 events

2590+3848+2201=8639

I think its correct...

Any suggestions are welcomed

Answer 2

If there are fields common to both event types then you can use a left join to combine the data. This is slow and subject to a limit of 50,000 results.

index=1idx1 sourcetype=src | dedup A 
| join type=outer A [search index=idx2 sourcetype=src | dedup A]
| ...

A more performant way to merge results is with the stats command, but that doesn't honor the requirement that index A trump index B.