org.apache.arrow.vector

Class BaseFixedWidthVector

java.lang.Object

org.apache.arrow.vector.BaseValueVector

org.apache.arrow.vector.BaseFixedWidthVector

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<ValueVector>, ElementAddressableVector, FieldVector, FixedWidthVector, ValueVector, VectorDefinitionSetter

Direct Known Subclasses:

BigIntVector, BitVector, DateDayVector, DateMilliVector, Decimal256Vector, DecimalVector, DurationVector, FixedSizeBinaryVector, Float2Vector, Float4Vector, Float8Vector, IntervalDayVector, IntervalMonthDayNanoVector, IntervalYearVector, IntVector, SmallIntVector, TimeMicroVector, TimeMilliVector, TimeNanoVector, TimeSecVector, TimeStampVector, TinyIntVector, UInt1Vector, UInt2Vector, UInt4Vector, UInt8Vector

public abstract class BaseFixedWidthVector
extends BaseValueVector
implements FixedWidthVector, FieldVector, VectorDefinitionSetter

BaseFixedWidthVector provides an abstract interface for implementing vectors of fixed width values. The vectors are nullable implying that zero or more elements in the vector could be NULL.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	actualValueCapacity
protected Field	field
protected int	lastValueCapacity
protected ArrowBuf	validityBuffer
protected ArrowBuf	valueBuffer
protected int	valueCount

Fields inherited from class org.apache.arrow.vector.BaseValueVector

allocator, fieldReader, INITIAL_VALUE_ALLOCATION, MAX_ALLOCATION_SIZE, MAX_ALLOCATION_SIZE_PROPERTY

Constructor Summary

Constructors

Constructor and Description
BaseFixedWidthVector(Field field, BufferAllocator allocator, int typeWidth) Constructs a new instance.

Method Summary

Modifier and Type	Method and Description
<OUT,IN> OUT	accept(VectorVisitor<OUT,IN> visitor, IN value) Accept a generic VectorVisitor and return the result.
void	allocateNew() Same as allocateNewSafe().
void	allocateNew(int valueCount) Allocate memory for the vector to support storing at least the provided number of elements in the vector.
boolean	allocateNewSafe() Allocate memory for the vector.
void	clear() Same as close().
void	close() Close the vector and release the associated buffers.
void	copyFrom(int fromIndex, int thisIndex, ValueVector from) Copy a cell value from a particular index in source vector to a particular position in this vector.
void	copyFromSafe(int fromIndex, int thisIndex, ValueVector from) Same as copyFrom(int, int, ValueVector) except that it handles the case when the capacity of the vector needs to be expanded before copy.
protected void	decrementAllocationMonitor()
ArrowBuf[]	getBuffers(boolean clear) Return the underlying buffers associated with this vector.
int	getBufferSize() Get the size (number of bytes) of underlying buffers used by this vector.
int	getBufferSizeFor(int count) Get the potential buffer size for a particular number of records.
List<FieldVector>	getChildrenFromFields() Get the inner child vectors.
ArrowBuf	getDataBuffer() Get the buffer that stores the data for elements in the vector.
long	getDataBufferAddress() Get the memory address of buffer that stores the data for elements in the vector.
ArrowBufPointer	getDataPointer(int index) Gets the pointer for the data at the given index.
ArrowBufPointer	getDataPointer(int index, ArrowBufPointer reuse) Gets the pointer for the data at the given index.
Field	getField() Get information about how this field is materialized.
List<ArrowBuf>	getFieldBuffers() Get the buffers belonging to this vector.
List<BufferBacked>	getFieldInnerVectors() Deprecated. This API will be removed as the current implementations no longer support inner vectors.
String	getName() Gets the name of the vector.
int	getNullCount() Get the number of elements that are null in the vector.
ArrowBuf	getOffsetBuffer() buffer that stores the offsets for elements in the vector.
long	getOffsetBufferAddress() Get the memory address of buffer that stores the offsets for elements in the vector.
TransferPair	getTransferPair(BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
abstract TransferPair	getTransferPair(Field field, BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
TransferPair	getTransferPair(Field field, BufferAllocator allocator, CallBack callBack) Construct a transfer pair of this vector and another vector of same type.
abstract TransferPair	getTransferPair(String ref, BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
TransferPair	getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) Construct a transfer pair of this vector and another vector of same type.
int	getTypeWidth()
ArrowBuf	getValidityBuffer() Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
long	getValidityBufferAddress() Get the memory address of buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
protected int	getValidityBufferValueCapacity()
protected int	getValueBufferValueCapacity()
int	getValueCapacity() Get the current value capacity for the vector.
int	getValueCount() Get the value count of vector.
protected void	handleSafe(int index)
int	hashCode(int index) Returns hashCode of element in index with the default hasher.
int	hashCode(int index, ArrowBufHasher hasher) Returns hashCode of element in index with the given hasher.
protected void	incrementAllocationMonitor()
void	initializeChildrenFromFields(List<Field> children) Initialize the children in schema for this Field.
boolean	isNull(int index) Check if element at given index is null.
boolean	isSafe(int index) Check if the given index is within the current value capacity of the vector.
int	isSet(int index) Same as isNull(int).
void	loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) Load the buffers of this vector with provided source buffers.
void	reAlloc() Resize the vector to increase the capacity.
protected void	refreshValueCapacity() Call this if you change the capacity of valueBuffer or validityBuffer.
void	reset() Reset the vector to initial state.
void	set(int index, byte[] value, int start, int length)
void	set(int index, ByteBuffer value, int start, int length)
void	setIndexDefined(int index) Mark the particular position in the vector as non-null.
void	setInitialCapacity(int valueCount) Sets the desired value capacity for the vector.
void	setNull(int index) Set the element at the given index to null.
void	setSafe(int index, byte[] value, int start, int length)
void	setSafe(int index, ByteBuffer value, int start, int length)
void	setValueCount(int valueCount) Set value count for the vector.
void	splitAndTransferTo(int startIndex, int length, BaseFixedWidthVector target) Slice this vector at desired index and length and transfer the corresponding data to the target vector.
void	transferTo(BaseFixedWidthVector target) Transfer this vector's data to another vector.
void	validateScalars() Validate the scalar values held by this vector.
void	zeroVector() zero out the vector and the data in associated buffers.

Methods inherited from class org.apache.arrow.vector.BaseValueVector

checkBufRefs, getAllocator, getReader, getReaderImpl, getValidityBufferSizeFromCount, iterator, releaseBuffer, toString, transferBuffer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.arrow.vector.ValueVector

getAllocator, getMinorType, getObject, getReader, makeTransferPair, validate, validateFull

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Methods inherited from interface org.apache.arrow.vector.FieldVector

exportBuffer, exportCDataBuffers

Field Detail

lastValueCapacity

protected int lastValueCapacity

actualValueCapacity

protected int actualValueCapacity

field

protected final Field field

validityBuffer

protected ArrowBuf validityBuffer

valueBuffer

protected ArrowBuf valueBuffer

valueCount

protected int valueCount

Constructor Detail

BaseFixedWidthVector

public BaseFixedWidthVector(Field field,
                            BufferAllocator allocator,
                            int typeWidth)

Constructs a new instance.

Parameters:

field - field materialized by this vector

allocator - The allocator to use for allocating memory for the vector.

typeWidth - The width in bytes of the type.

Method Detail

getTypeWidth

public int getTypeWidth()

getName

public String getName()

Description copied from interface: ValueVector

Gets the name of the vector.

Specified by:

getName in interface ValueVector

Specified by:

getName in class BaseValueVector

Returns:

the name of the vector.

getValidityBufferAddress

public long getValidityBufferAddress()

Get the memory address of buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.

Specified by:

getValidityBufferAddress in interface FieldVector

Returns:

starting address of the buffer

getDataBufferAddress

public long getDataBufferAddress()

Get the memory address of buffer that stores the data for elements in the vector.

Specified by:

getDataBufferAddress in interface FieldVector

Returns:

starting address of the buffer

getOffsetBufferAddress

public long getOffsetBufferAddress()

Get the memory address of buffer that stores the offsets for elements in the vector. This operation is not supported for fixed-width vectors.

Specified by:

getOffsetBufferAddress in interface FieldVector

Returns:

starting address of the buffer

Throws:

UnsupportedOperationException - for fixed width vectors

getValidityBuffer

public ArrowBuf getValidityBuffer()

Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector. Consider it as a buffer for internal bit vector data structure.

Specified by:

getValidityBuffer in interface ValueVector

Returns:

buffer

getDataBuffer

public ArrowBuf getDataBuffer()

Get the buffer that stores the data for elements in the vector.

Specified by:

getDataBuffer in interface ValueVector

Returns:

buffer

getOffsetBuffer

public ArrowBuf getOffsetBuffer()

buffer that stores the offsets for elements in the vector. This operation is not supported for fixed-width vectors.

Specified by:

getOffsetBuffer in interface ValueVector

Returns:

buffer

Throws:

UnsupportedOperationException - for fixed width vectors

setInitialCapacity

public void setInitialCapacity(int valueCount)

Sets the desired value capacity for the vector. This function doesn't allocate any memory for the vector.

Specified by:

setInitialCapacity in interface ValueVector

Parameters:

valueCount - desired number of elements in the vector

getValueCapacity

public int getValueCapacity()

Get the current value capacity for the vector.

Specified by:

getValueCapacity in interface ValueVector

Returns:

number of elements that vector can hold.

refreshValueCapacity

protected void refreshValueCapacity()

Call this if you change the capacity of valueBuffer or validityBuffer.

getValueBufferValueCapacity

protected int getValueBufferValueCapacity()

getValidityBufferValueCapacity

protected int getValidityBufferValueCapacity()

zeroVector

public void zeroVector()

zero out the vector and the data in associated buffers.

Specified by:

zeroVector in interface FixedWidthVector

reset

public void reset()

Reset the vector to initial state. Same as zeroVector(). Note that this method doesn't release any memory.

Specified by:

reset in interface ValueVector

close

public void close()

Close the vector and release the associated buffers.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ValueVector

Overrides:

close in class BaseValueVector

clear

public void clear()

Same as close().

Specified by:

clear in interface ValueVector

Overrides:

clear in class BaseValueVector

incrementAllocationMonitor

protected void incrementAllocationMonitor()

decrementAllocationMonitor

protected void decrementAllocationMonitor()

allocateNew

public void allocateNew()

Same as allocateNewSafe().

Specified by:

allocateNew in interface ValueVector

allocateNewSafe

public boolean allocateNewSafe()

Allocate memory for the vector. We internally use a default value count of 4096 to allocate memory for at least these many elements in the vector. See allocateNew(int) for allocating memory for specific number of elements in the vector.

Specified by:

allocateNewSafe in interface ValueVector

Returns:

false if memory allocation fails, true otherwise.

allocateNew

public void allocateNew(int valueCount)

Allocate memory for the vector to support storing at least the provided number of elements in the vector. This method must be called prior to using the ValueVector.

Specified by:

allocateNew in interface FixedWidthVector

Parameters:

valueCount - the desired number of elements in the vector

Throws:

OutOfMemoryException - on error

getBufferSizeFor

public int getBufferSizeFor(int count)

Get the potential buffer size for a particular number of records.

Specified by:

getBufferSizeFor in interface ValueVector

Parameters:

count - desired number of elements in the vector

Returns:

estimated size of underlying buffers if the vector holds a given number of elements

getBufferSize

public int getBufferSize()

Get the size (number of bytes) of underlying buffers used by this vector.

Specified by:

getBufferSize in interface ValueVector

Returns:

size of underlying buffers.

getField

public Field getField()

Get information about how this field is materialized.

Specified by:

getField in interface ValueVector

Returns:

the field corresponding to this vector

getBuffers

public ArrowBuf[] getBuffers(boolean clear)

Return the underlying buffers associated with this vector. Note that this doesn't impact the reference counts for this buffer so it only should be used for in-context access. Also note that this buffer changes regularly thus external classes shouldn't hold a reference to it (unless they change it).

Specified by:

getBuffers in interface ValueVector

Parameters:

clear - Whether to clear vector before returning; the buffers will still be refcounted but the returned array will be the only reference to them

Returns:

The underlying buffers that is used by this vector instance.

reAlloc

public void reAlloc()

Resize the vector to increase the capacity. The internal behavior is to double the current value capacity.

Specified by:

reAlloc in interface ValueVector

getFieldInnerVectors

@Deprecated
public List<BufferBacked> getFieldInnerVectors()

Deprecated. This API will be removed as the current implementations no longer support inner vectors.

Get the inner vectors.

Specified by:

getFieldInnerVectors in interface FieldVector

Returns:

the inner vectors for this field as defined by the TypeLayout

initializeChildrenFromFields

public void initializeChildrenFromFields(List<Field> children)

Initialize the children in schema for this Field. This operation is a NO-OP for scalar types since they don't have any children.

Specified by:

initializeChildrenFromFields in interface FieldVector

Parameters:

children - the schema

Throws:

IllegalArgumentException - if children is a non-empty list for scalar types.

getChildrenFromFields

public List<FieldVector> getChildrenFromFields()

Get the inner child vectors.

Specified by:

getChildrenFromFields in interface FieldVector

Returns:

list of child vectors for complex types, empty list for scalar vector types

loadFieldBuffers

public void loadFieldBuffers(ArrowFieldNode fieldNode,
                             List<ArrowBuf> ownBuffers)

Load the buffers of this vector with provided source buffers. The caller manages the source buffers and populates them before invoking this method.

Specified by:

loadFieldBuffers in interface FieldVector

Parameters:

fieldNode - the fieldNode indicating the value count

ownBuffers - the buffers for this Field (own buffers only, children not included)

getFieldBuffers

public List<ArrowBuf> getFieldBuffers()

Get the buffers belonging to this vector.

Specified by:

getFieldBuffers in interface FieldVector

Returns:

the inner buffers.

validateScalars

public void validateScalars()

Validate the scalar values held by this vector.

getTransferPair

public TransferPair getTransferPair(String ref,
                                    BufferAllocator allocator,
                                    CallBack callBack)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

ref - name of the target vector

allocator - allocator for the target vector

callBack - not used

Returns:

TransferPair

getTransferPair

public TransferPair getTransferPair(Field field,
                                    BufferAllocator allocator,
                                    CallBack callBack)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

field - The field materialized by this vector.

allocator - allocator for the target vector

callBack - not used

Returns:

TransferPair

getTransferPair

public TransferPair getTransferPair(BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Overrides:

getTransferPair in class BaseValueVector

Parameters:

allocator - allocator for the target vector

Returns:

TransferPair

getTransferPair

public abstract TransferPair getTransferPair(String ref,
                                             BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

ref - name of the target vector

allocator - allocator for the target vector

Returns:

TransferPair

getTransferPair

public abstract TransferPair getTransferPair(Field field,
                                             BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

field - Field object used by the target vector

allocator - allocator for the target vector

Returns:

TransferPair

transferTo

public void transferTo(BaseFixedWidthVector target)

Transfer this vector's data to another vector. The memory associated with this vector is transferred to the allocator of target vector for accounting and management purposes.

Parameters:

target - destination vector for transfer

splitAndTransferTo

public void splitAndTransferTo(int startIndex,
                               int length,
                               BaseFixedWidthVector target)

Slice this vector at desired index and length and transfer the corresponding data to the target vector.

Parameters:

startIndex - start position of the split in source vector.

length - length of the split.

target - destination vector

getNullCount

public int getNullCount()

Get the number of elements that are null in the vector.

Specified by:

getNullCount in interface ValueVector

Returns:

the number of null elements.

getValueCount

public int getValueCount()

Get the value count of vector. This will always be zero unless setValueCount(int) has been called prior to calling this.

Specified by:

getValueCount in interface ValueVector

Returns:

valueCount for the vector

setValueCount

public void setValueCount(int valueCount)

Set value count for the vector.

Specified by:

setValueCount in interface ValueVector

Parameters:

valueCount - value count to set

isSafe

public boolean isSafe(int index)

Check if the given index is within the current value capacity of the vector.

Parameters:

index - position to check

Returns:

true if index is within the current value capacity

isNull

public boolean isNull(int index)

Check if element at given index is null.

Specified by:

isNull in interface ValueVector

Parameters:

index - position of element

Returns:

true if element at given index is null, false otherwise

isSet

public int isSet(int index)

Same as isNull(int).

Parameters:

index - position of element

Returns:

1 if element at given index is not null, 0 otherwise

setIndexDefined

public void setIndexDefined(int index)

Mark the particular position in the vector as non-null.

Specified by:

setIndexDefined in interface VectorDefinitionSetter

Parameters:

index - position of the element.

set

public void set(int index,
                byte[] value,
                int start,
                int length)

setSafe

public void setSafe(int index,
                    byte[] value,
                    int start,
                    int length)

set

public void set(int index,
                ByteBuffer value,
                int start,
                int length)

setSafe

public void setSafe(int index,
                    ByteBuffer value,
                    int start,
                    int length)

handleSafe

protected void handleSafe(int index)

copyFrom

public void copyFrom(int fromIndex,
                     int thisIndex,
                     ValueVector from)

Copy a cell value from a particular index in source vector to a particular position in this vector. The source vector should be of the same type as this one.

Specified by:

copyFrom in interface ValueVector

Overrides:

copyFrom in class BaseValueVector

Parameters:

fromIndex - position to copy from in source vector

thisIndex - position to copy to in this vector

from - source vector

copyFromSafe

public void copyFromSafe(int fromIndex,
                         int thisIndex,
                         ValueVector from)

Same as copyFrom(int, int, ValueVector) except that it handles the case when the capacity of the vector needs to be expanded before copy.

Specified by:

copyFromSafe in interface ValueVector

Overrides:

copyFromSafe in class BaseValueVector

Parameters:

fromIndex - position to copy from in source vector

thisIndex - position to copy to in this vector

from - source vector

setNull

public void setNull(int index)

Set the element at the given index to null.

Specified by:

setNull in interface FieldVector

Parameters:

index - position of element

getDataPointer

public ArrowBufPointer getDataPointer(int index)

Description copied from interface: ElementAddressableVector

Gets the pointer for the data at the given index.

Specified by:

getDataPointer in interface ElementAddressableVector

Parameters:

index - the index for the data.

Returns:

the pointer to the data.

getDataPointer

public ArrowBufPointer getDataPointer(int index,
                                      ArrowBufPointer reuse)

Description copied from interface: ElementAddressableVector

Gets the pointer for the data at the given index.

Specified by:

getDataPointer in interface ElementAddressableVector

Parameters:

index - the index for the data.

reuse - the data pointer to fill, this avoids creating a new pointer object.

Returns:

the pointer to the data, it should be the same one as the input parameter

hashCode

public int hashCode(int index)

Description copied from interface: ValueVector

Returns hashCode of element in index with the default hasher.

Specified by:

hashCode in interface ValueVector

hashCode

public int hashCode(int index,
                    ArrowBufHasher hasher)

Description copied from interface: ValueVector

Returns hashCode of element in index with the given hasher.

Specified by:

hashCode in interface ValueVector

accept

public <OUT,IN> OUT accept(VectorVisitor<OUT,IN> visitor,
                           IN value)

Description copied from interface: ValueVector

Accept a generic VectorVisitor and return the result.

Specified by:

accept in interface ValueVector

Type Parameters:

OUT - the output result type.

IN - the input data together with visitor.