Damaged nc-file

[ziand]

Hallo, I did several subsequent calculations and wrote results to one big nc-file. My object_id's look like this:

Code

object_id= 'angle' + str(int(angle)).rjust(2, '0')

Angle is 0.0, 1.0, ... so I get object id's like 'angle-000', 'angle-001', ... My nc-file is very big ~4.3GB. When I wanted to read a configuration I got the following error

  File "./zipdir/NL/IO/NLSaveUtilities.py", line 252, in nlread   File "./zipdir/NL/IO/NetCDFFile.py", line 156, in __init__   File "./zipdir/NL/IO/NetCDFFile.py", line 369, in _read   File "./zipdir/NL/IO/NetCDFFile.py", line 440, in _read_var_array

If a have a wrong object_id, the output of nlread is an empty array []. But not such an error. Did you see someting like this before? Could it be related to the file-size being too large? Normally VNL is able to quickly print the object_id's of an nc-file. But it does not work here. Is there a way to find out the object-id's manually with atkpython? Is this file unrepairably damaged or could there be a way to still get some data out of it?

[ziand]

I made a test within atkpython (interactive mode) and got basically the same error:

In [14]: x=nlread('Cu_5x5x5_CNT_6x0x2_Geom_185.nc',object_id='angle-000')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
/home/ziand/atk/CNTDevice/src/<ipython console> in <module>()
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NLSaveUtilities.pyc in nlread(filename, class_type, object_id, read_state)
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NetCDFFile.pyc in __init__(self, filename, mode, mmap)
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NetCDFFile.pyc in _read(self)
/opt/QuantumWise/atk-10.8.0/atkpython/lib/python26.zip/NL/IO/NetCDFFile.pyc in _read_var_array(self)
IOError: [Errno 22] Invalid argument

[Nordland]

The following code snippet gives the list of object id's in the file.

Code: python

from NL.IO.NLSaveUtilities import nlinspect
print nlinspect('your_file.nc')

However I think there is 2 possibilities that can cause this problem.

• The file is broken - caused by an interruption in saving or so - if you install the program ncdump and is able to see inside the file, it is likely that it is not broken.
• If the file is not broken, then file might be too large to read in. This can either be an issue in the code or it can be related to memory on the machine.

[Anders Blom]

"ncdump" is part of the NetCDF package from Unidata.

For Linux it can be obtained as pre-compiled binaries from http://www.unidata.ucar.edu/downloads/netcdf/index.jsp and for Windows at http://www.unidata.ucar.edu/software/netcdf/docs/faq.html (search for "Windows" on the page). Or, you can try to built it from source (http://www.unidata.ucar.edu/downloads/netcdf/index.jsp), if you have all the required libraries.

[ziand]

Thank you very much. The code for printing the object id's can be quite helpful with big files.

About my problem: Only one of my files is broken. The others are okay, I myself made a mistake there. So the problem is not very big for me. Anyway a installed ncdump and indeed was able to look inside the broken file. But before it gave me an error. After pressing "Ignore" two times it started dumping the file. The error message was on a small window from visual studio c++ runtime library:

Assertion failed!
Program ncdump
File: ..\..\..\libsrc\v1hpg.c
Line: 1157
Expression: ncp->begin_rec > 0

However, don't care about it as long as it does not happen more often. For now I have no problem here  .