how to build pipeline_kafka extension

Question

i'm currently trying to compile the pipeline_kafka extension from Github to add it to my pipelinedb instance but i'm running in a problem neither i nor google seems to be able to solve.

when i execute make command i get the following error

stc@vm:~/pipeline_kafka$ make gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.6 -c -o pipeline_kafka.o pipeline_kafka.c pipeline_kafka.c:23:40: fatal error: catalog/pipeline_stream_fn.h: No such file or directory #include "catalog/pipeline_stream_fn.h" ^ compilation terminated. : recipe for target 'pipeline_kafka.o' failed make: *** [pipeline_kafka.o] Error 1

does anyone has an idea what to do? I'm running on debian 8.

Does anyone know why this extension is not part of pipelinedb anymore?

cheers and thanks in advance

Answer 1

the reason for the error is my entire approach towards using/installing pipelinedb and pipeline_kafka.

the following script worked for me on a standard Debian 8 installation. Maybe it helps.

Download and install PipelineDB

wget -O pipelinedb_install_package "https://www.pipelinedb.com/download/0.9.1/debian8"

sudo dpkg -i pipelinedb_install_package

mkdir ~/logs

pipeline-init -D ~/data/pipelinedb
pipeline-ctl -D ~/data/pipelinedb -l ~/logs/pipelinedb.log start
sleep 5.0
pipeline pipeline 

Download and install pipeline_kafka extension

sudo apt-get install git gcc g++ zlib1g-dev

mkdir ~/git

git clone https://github.com/edenhill/librdkafka.git ~/git/librdkafka
git clone https://github.com/pipelinedb/pipeline_kafka.git  >~/git/pipeline_kafka

cd ~/git/librdkafka
./configure --prefix=/usr 
make
sudo make install

cd ~/git/pipeline_kafka
./configure 
make
sudo make install

pipeline pipeline -c "CREATE EXTENSION pipeline_kafka;"

sed -i 's/#shared_preload_libraries/shared_preload_libraries/' ~/data/pipelinedb/pipelinedb.conf
sed -i "s/shared_preload_libraries = ''/shared_preload_libraries=pipeline_kafka/" ~/data/pipelinedb/pipelinedb.conf