Remove null fields in JSON String in Perl

Question

I am modifying a Perl script that collect some data from various sources, put them into a structured JSON and call a web service passing that JSON file. I have a problem with null values in some of these fields ...

This is a snippet of my code:

  $p->{BAND_ATTEN_DN1} = 'null';
  $p->{BAND_ATTEN_DN2} = 'null';
  $p->{BAND_ATTEN_DN3} = 'null';
  $p->{BAND_ATTEN_DN4} = 'null';
  $p->{BAND_ATTEN_UP0} = 'null';
  $p->{BAND_ATTEN_UP1} = 'null';
  $p->{BAND_ATTEN_UP2} = 'null';
  $p->{BAND_ATTEN_UP3} = 'null';
  $p->{BAND_ATTEN_UP4} = 'null';
  $p->{BAND_SNR_MARGIN_DN1} = 'null';
  $p->{BAND_SNR_MARGIN_DN2} = 'null';
  $p->{BAND_SNR_MARGIN_DN3} = 'null';
  $p->{BAND_SNR_MARGIN_DN4} = 'null';
  $p->{BAND_SNR_MARGIN_UP0} = 'null';
  $p->{BAND_SNR_MARGIN_UP1} = 'null';
  $p->{BAND_SNR_MARGIN_UP2} = 'null';
  $p->{BAND_SNR_MARGIN_UP3} = 'null';
  $p->{BAND_SNR_MARGIN_UP4} = 'null';
  $p->{BAND_SIG_ATTEN_DN1} = 'null';
  $p->{BAND_SIG_ATTEN_DN2} = 'null';
  $p->{BAND_SIG_ATTEN_DN3} = 'null';
  $p->{BAND_SIG_ATTEN_DN4}  = 'null';

  $p->{BAND_SIG_ATTEN_UP0} = 'null';
  $p->{BAND_SIG_ATTEN_UP1} = 'null';
  $p->{BAND_SIG_ATTEN_UP2} = 'null';
  $p->{BAND_SIG_ATTEN_UP3} = 'null';
  $p->{BAND_SIG_ATTEN_UP4} = 'null';
  $p->{ATTAINABLE_DN} = 'null';
  $p->{ATTAINABLE_UP} = 'null';

  $p->{POWER_DN} = 'null';
  $p->{POWER_UP} = 'null';

  for my $k (keys %$q) {
    $p->{$k} =  ($q->{$k}) ? $q->{$k} : 'null';
  }

  my $m = $measure->{$port};
  for my $k (keys %$m) {
    $p->{$k} =  ($m->{$k}) ? $m->{$k} : 'null';
  }

  my $hlog_up_values = &get_values($delt->{UPHLOG}, 9999, -99);
  my $hlog_dn_values = &get_values($delt->{DOWNHLOG}, 9999, -99);
  my $qln_up_values = &get_values($delt->{UPQLN}, 9999, -199);
  my $qln_dn_values = &get_values($delt->{DOWNQLN}, 9999, -199);
  my $snr_up_values = &get_values($delt->{UPSNR}, 9999, -99);
  my $snr_dn_values = &get_values($delt->{DOWNSNR}, 9999, -99);

  my $hlog_cg_size_dn = grep ($delt->{DOWNHLOGGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{DOWNHLOGGRUOPSIZE} : $delt->{DOWNHLOGGRUOPSIZE}/4.3125;
  my $hlog_cg_size_up = grep ($delt->{UPHLOGGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{UPHLOGGRUOPSIZE} : $delt->{UPHLOGGRUOPSIZE}/4.3125;
  my $qln_cg_size_dn = grep ($delt->{DOWNQLNGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{DOWNQLNGRUOPSIZE} : $delt->{DOWNQLNGRUOPSIZE}/4.3125;
  my $qln_cg_size_up = grep ($delt->{UPQLNGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{UPQLNGRUOPSIZE} :$delt->{UPQLNGRUOPSIZE}/4.3125;
  my $snr_cg_size_dn = grep ($delt->{DOWNSNRGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{DOWNSNRGRUOPSIZE} :$delt->{DOWNSNRGRUOPSIZE}/4.3125;
  my $snr_cg_size_up = grep ($delt->{UPSNRGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{UPSNRGRUOPSIZE} : $delt->{UPSNRGRUOPSIZE}/4.3125;

  my $pots_line_length_A    = defined $plt->{POTS_LINE_LENGTH_A} ? $plt->{POTS_LINE_LENGTH_A} : 'null';
  my $pots_line_length_B    = defined $plt->{POTS_LINE_LENGTH_B} ? $plt->{POTS_LINE_LENGTH_B} : 'null';
  my $pots_line_state       = defined $plt->{POTS_LINE_STATE} ? $plt->{POTS_LINE_STATE} : 'null';
  my $pots_line_termination = defined $plt->{POTS_LINE_TERMINATION_TYPE} ? $plt->{POTS_LINE_TERMINATION_TYPE} : 'null';


  #&_trace("STUB GROUP SIZE",$function, TRC_PDNT);
  #$hlog_cg_size_dn = 8;
  #$hlog_cg_size_up = 8;
  #$qln_cg_size_dn = 8;
  #$qln_cg_size_up = 8;
  #$snr_cg_size_dn = 8;
  #$snr_cg_size_up = 8;


  my $json_string = "{
    \"attainable_rate\": {
      \"down_value\": $p->{ATTAINABLE_DN},
      \"up_value\": $p->{ATTAINABLE_UP}
    },
    \"hlog\": {
      \"up_values\": [$hlog_up_values],
      \"down_values\": [$hlog_dn_values]
    },
    \"hlog_cg_size\": {
      \"down_value\": $hlog_cg_size_dn,
      \"up_value\": $hlog_cg_size_up
    },
    \"qln\": {
      \"down_values\": [$qln_up_values],
      \"up_values\": [$qln_dn_values]
    },
    \"qln_cg_size\": {
      \"down_value\": $qln_cg_size_dn,
      \"up_value\": $qln_cg_size_up
    },
    \"snr\": {
      \"up_values\": [$snr_up_values],
      \"down_values\": [$snr_dn_values]
    },
    \"snr_cg_size\": {
      \"down_value\": $snr_cg_size_dn,
      \"up_value\": $snr_cg_size_up
    },
    \"spectrum\": \"$actual_spectrum\",

    \"rtx_status\": {
      \"down_value\": [
        null
      ],
      \"up_value\": [
        null
      ]
    },
    \"line_attenuation\": {
      \"down_values\": [
        $p->{BAND_ATTEN_DN1},
        $p->{BAND_ATTEN_DN2},
        $p->{BAND_ATTEN_DN3},
        $p->{BAND_ATTEN_DN4}
      ],
      \"up_values\": [
        $p->{BAND_ATTEN_UP0},
        $p->{BAND_ATTEN_UP1},
        $p->{BAND_ATTEN_UP2},
        $p->{BAND_ATTEN_UP3},
        $p->{BAND_ATTEN_UP4}
      ]
    },
    \"noise_margin\": {
      \"down_values\": [
        if($p->{BAND_SNR_MARGIN_DN1},
        $p->{BAND_SNR_MARGIN_DN2},
        $p->{BAND_SNR_MARGIN_DN3},
        $p->{BAND_SNR_MARGIN_DN4}
      ],
      \"up_values\": [
        $p->{BAND_SNR_MARGIN_UP0},
        $p->{BAND_SNR_MARGIN_UP1},
        $p->{BAND_SNR_MARGIN_UP2},
        $p->{BAND_SNR_MARGIN_UP3},
        $p->{BAND_SNR_MARGIN_UP4}
      ]
    },
    \"signal_attenuation\": {
      \"down_values\": [
       $p->{BAND_SIG_ATTEN_DN1},
       $p->{BAND_SIG_ATTEN_DN2},
       $p->{BAND_SIG_ATTEN_DN3},
       $p->{BAND_SIG_ATTEN_DN4}
      ],
      \"up_values\": [
       $p->{BAND_SIG_ATTEN_UP0},
       $p->{BAND_SIG_ATTEN_UP1},
       $p->{BAND_SIG_ATTEN_UP2},
       $p->{BAND_SIG_ATTEN_UP3},
       $p->{BAND_SIG_ATTEN_UP4}
      ]
    },
    \"transmit_power\": {
      \"down_value\": $p->{POWER_DN},
      \"up_value\": $p->{POWER_UP}
    }
  }";

For example (I know for sure the problem is present only for line_attenuation, signal_attenuation and noise_margin both up and down)

   \"line_attenuation\": {
      \"down_values\": [
        $p->{BAND_ATTEN_DN1},
        $p->{BAND_ATTEN_DN2},
        $p->{BAND_ATTEN_DN3},
        $p->{BAND_ATTEN_DN4}

if $p->{BAND_ATTEN_DN4} is null I have to pass only the first 3 values and not 3 valid value+1 null .. how can I delete those null values? checking all those fields?

Currently I'm using those library

use strict;
use Data::Dumper;
use Getopt::Long;
use REST::Client;
use DBI;
use JSON;

If it is possible I would like not to import other libraries or so.

I have tryed checking if defined, or putting if in the json code but nothing ...

Thanks for the help

Answer 1

The smart way to do this would be to use the JSON module to convert your data structure to JSON. I would either build a new Perl data structure up step by step

my $to_json;

$to_json->{foo} = $p->{foo};

if ($p->{bar}) {
    $to_json->{baz} = $p->{baz};
}

or use inline operations with map and grep.

use strict;
use warnings;
use JSON;

my $p = {
    BAND_ATTEN_DN1 => 1,
    BAND_ATTEN_DN2 => 2,
    BAND_ATTEN_DN3 => 3,
    BAND_ATTEN_DN4 => undef,
};

my $to_json = {
    line_attenuaion => {
        down_values => [
            grep defined, map { $p->{$_} } 
                qw/BAND_ATTEN_DN1 BAND_ATTEN_DN2 BAND_ATTEN_DN3 BAND_ATTEN_DN4/,
        ],
        hashref_slice => [
            grep defined,
                @{$p}{qw/BAND_ATTEN_DN1 BAND_ATTEN_DN2 BAND_ATTEN_DN3 BAND_ATTEN_DN4/},
        ],
    },
    # ...
};

print to_json $to_json;

---

However, you say you cannot make large modifications. You'll have to split up the code that outputs the JSON instead. This is going to produce even uglier code and I strongly recommend not to do it this way, for the sake of maintenance.

# HERE BE DRAGONS...

my $json_string = "{
    \"attainable_rate\": {
        \"down_value\": $p->{ATTAINABLE_DN},
        \"up_value\": $p->{ATTAINABLE_UP}
    },
    \"line_attenuation\": {
        \"down_values\": [
            $p->{BAND_ATTEN_DN1},
            $p->{BAND_ATTEN_DN2},
            $p->{BAND_ATTEN_DN3}";
if ($p->{BAND_ATTEN_DN4}) {
    $json_string .= ",
            $p->{BAND_ATTEN_DN4}";
}
$json_string .= "
        ]
    }
}
";

If you do this, then at least get rid of all the escaping by converting your double quotes "" for the string to either the qq operator or by using HEREDOC.

my $json_string = qq({
    "attainable_rate": {
        "down_value": $p->{ATTAINABLE_DN},
        "up_value": $p->{ATTAINABLE_UP}
    }
}
);

my $heredoc = <<JSON
{
    "attainable_rate": {
        "down_value": $p->{ATTAINABLE_DN},
        "up_value": $p->{ATTAINABLE_UP}
    }
}
JSON